A Managers Diary

Software engineering as a profession is fun. The role shift from a software engineer to a people manager in quite common place in Indian IT industry. Managing people brings forth its own set of challenges. For those with a MBA degree, the transition is relatively smooth. For others, the opportunity requires climbing the learning curve. I believe I’ve learned a thing or two from others experiences and my own experiences in managing people. I hope you find them useful.

1)  Be empathetic: You need to be able to put yourself in other people’s shoes, to see things from their perspective, and have an idea of what you would do if you where them. One way is to listen to your engineers actively without talking, understanding their concerns or ask questions that allow them to tell you more about themselves instead of yes and no questions.

2) Communication - Lifeblood of a project: One of the major contributors to a project's success is communication within and outside the team. An average person is not a great communicator. In particular, a significant number of gifted software developers are introverts. Sometimes they have a lot going on inside, and little time to express it to others. As a manager you must overcome this obstacle by actively connecting with people, and helping them connect with each other. It takes effort to get communication going, and once the momentum is built up, things become much easire.

3) There’s always a right thing to do, and sometimes it’s uncomfortable: For instance, there are things at work that I would have done differently if I had been designing or coding a particular component. This doesn’t mean they would have turned out better or worse.

Last year, while I was busy on a project, my team was ready for another product release. The release bombed and we had to rollback. When I reconnected with my team to sync up on deployment architecture, they explained a number of deployment decisions they had made. My knee-jerk reaction to some of them was "this is not right". After thinking for a minute it was clear that they had done mostly the right things given their time constraints and resources.

Trusting people to do the right thing is hard, although it gets easier when your team is absolutely awesome. As a manager you have to let people do things that take you out of your comfort zone. That’s fine, because they must own their work.

The point I was trying to make is that a stitch in time saves nine. Don’t succumb to the temptation of doing the easy thing, because many times it will cost you in the short and long term.

4) Help others succeed: One of the most rewarding aspects of my job has been to see people grow and evolve. It feels great when you see someone who was a shy programmer fresh out of school become a great presenter who can design a system and explain it to others. Or when you can help someone get to the next level and share some of your experiences with them.

As a manager,  you have to continously challenge your team by giving them variety of problems and providing guidance. Let them evaluate the pros and cons and make an informed decision. Remember engineere have their own way of doing things. Often they will take shortcuts during development or make a quick fix and move on or simply reinvent the wheel. And such practices comes to haunt later when things do not work as expected in production or clients abolutely hate the application.

Yes, there will be some horrible mistakes and some stellar success. Helping engineers understand their mistakes, and guiding them in the right direction instead of being critical is one way to help them succeed.

It is something you have to genuinely want.

5) Be an umbrella: An important part of your job is to simply let people do their work. You report to people (your own boss, customers, shareholders) who want different things. In some cases they are contradictory, or change too often. Sometimes there are storms brewing in the upper atmosphere,  and which could be averted. There is no need to burden your team with constantly changing weather they can do nothing about. Shield your people. Let them stand under you umbrella.

6) Be firm but nice: You know that you have the authority to call the shots. That is a powerful weapon, and you must use it wisely and sparingly. It’s much better when you can motivate people to agree on what to do. It’s true that it doesn’t always happen, but that should be your goal.

With valuable inputs from my friends and colleagues. Please feel free to share your experiences.

Your review :)

BigMemory: Scaling vertically

Until recently, Moore's Law resulted in faster CPUs, but physical constraints - heat dissipation, for example - and computer requirements force manufacturers to place multiple cores to single CPU wafers. Increases in memory, however, are unconstrained by this type of physical requirement. For instance, today you can purchase standard Von Neumann servers from Oracle, Dell and HP with up to 2 TB of physical RAM and 64 cores. Servers with 32 cores and 512 GB of RAM are certainly more typical, but it's clear that today's commodity servers are now “big iron” in their own right.

The following table shows the random access times for different storage technologies:

Storage Technology	Latency
Registers	1-3ns
CPU L1 Cache	2-8ns
CPU L2 Cache	5-12ns
Memory (RAM)	10-60ns
High-speed network gear	10,000-30,000ns
Solid State Disk (SSD) Drives	70,000-120,000ns
Hard Disk Drives	3,000,000-10,000,000ns

Since most enterprise applications tend to be I/O bound (i.e. they spend too much time waiting for data stored on disk), it follows that these applications would benefit greatly from the use of the lower-latency forms of storage at the top of this hierarchy. Specifically, today's time-sensitive enterprise applications would speed up significantly without much modification if they could replace all disk access with memory usage.

To drive this point home further, note that with modern network technology, latencies are at worst around 10,000-30,000 ns, with even lower latencies and higher speeds possible. This means that with the right equipment, accessing memory on other servers over the network is still much faster than reading from a local hard disk drive. All of this proves that as an enterprise architect or developer, your goal should be to use as much memory as possible in your applications.

The original Java language and platform design took into account the problems developers had when manually managing memory with other languages. For instance, when memory management goes wrong, developers experience memory leaks (lack of memory de-allocation) or memory access violations due to accessing memory that has already been de-allocated or attempting to de-allocate memory more than once. To relieve developers of these potential problems, Java implemented automatic memory management of the Java heap with a garbage collector (GC). When a running program no longer requires specific objects, the Java garbage collector reclaims its memory within the Java heap. Memory management is no longer an issue for Java developers, which results in greater productivity overall.

Garbage collection works reasonably well, but it becomes increasingly stressed as the size of the Java heap and numbers of live objects within it increase. Today, GC works well with an occupied Java heap around 3-4 GB in size, which also just happens to be the 32-bit memory limit.

The size limits imposed by Java garbage collection explain why 64-bit Java use remains a minority despite the availability of commodity 64-bit CPUs, operating systems and Java for half a decade. Attempts in Java to consume a heap beyond 3-4 GB in size can result in large garbage collection pauses (where application threads are stopped so that the GC can reclaim dead objects), unpredictable application response times and large latencies that can violate your application's service level agreements. With large occupied Java heaps, it's not uncommon to experience multi-second pauses, often at the most inopportune moments.

Solving the Java Heap/GC Problem

Large heaps are desirable in cases such as in-process caching and sessions storage. Both of these use cases use a map-like API where a framework allocates and de-allocates resources programmatically with puts and removes, opening up a way to constrain and solve the garbage collection problem.

BigMemory implementation from Terracotta and Apache (incubated) is an all-Java implementation built on Java's advanced NIO technology.  BigMemory is just a hair slower than Java heap. Its in process with the JVM so there is no management complexity and it is pure Java so there is no deployment complexity or sensitivity to JVM version. It creates a cache store in memory but outside the Java heap using Direct Byte Buffers. By storing data off heap, the garbage collector does not know about it and therefore does not collect it. Instead, BigMemory responds to the put and remove requests to allocate and free memory in its managed byte buffer.

This lets you keep the Java heap relatively small (1-2GB in size), while using the maximum amount of objects within physical memory. As a result, BigMemory can create caches in memory that match physical RAM limits (i.e. 2TB today and more in the future), without the garbage collection penalties that usually come with a Java heap of that size. By storing your application's data outside of the Java heap but within RAM inside your Java process, you get all the benefits of in-memory storage without the traditional Java costs.

How does ByteBuffer help?

Prior to JDK 1.4, Java programmers had limited options: they could read data into a byte[] and use explicit offsets (along with bitwise operators) to combine bytes into larger entities, or they could wrap the byte[] in a DataInputStream and get automatic conversion without random access.

The ByteBuffer class arrived in JDK 1.4 as part of the java.nio package, and combines larger-than-byte data operations with random access. To construct a simple cache using ByteBuffer, see this article.

For those looking for in-depth explanation on the topic, read the article here. It is a long read but it is worth the information gain.

Note: Essentially, both these products are managing contiguous region of memory. Even though  the approach described above avoids GC, fragmentation in any contiguous region eventually has a cost. The compaction cycle would happen far less often than a JVM garbage collection cycle would, so while it would cruelly affect performance during the cycle, it would occur fairly rarely.

That brings up another topic: how does the non-heap memory for direct buffers get released? After all, there's no method to explicitly close or release them. The answer is that they get garbage collected like any other object, but with one twist: if you don't have enough virtual memory space or commit charge to allocate a direct buffer, that will trigger a full collection even if there's plenty of heap memory available. -

Conclusion

It still makes sense to scale horizontally. Even so, you still leverage vertical scalability with BigMemory, which makes the distributed cache faster with higher density.

Further reading
http://raffaeleguidi.wordpress.com/

Humor: Software

Software engineer 1: I am not able to connect to test database.
Please check the problem and restart it.

IT:  Issue resolved

Software engineer 2: What was the fix?

Software engineer 3:  Exorcism…





Keep smiling!
You may also like: Thirst

Getting started with HTML5 WebSockets and Java - Part 1

Any technology around HTML5 seems to be a hot button topic these days and lucky for me that I got an opportunity to take a deep dive into WebSockets. Be it canvas, geolocation, video playback, drag-and-drop or WebSocket , there is a lot of buzz around these upcoming technologies.

Some background on HTML5 WebSockets

HTML5 WebSocket defines a bi-directional, full-duplex communication channel operates through a single TCP connection. The important thing to note is the WebSocket API is being standardized by the W3C, and the WebSocket protocol has been standardized by the IETF as RFC 6455.

What this means is that there are bunch of protocol versions and today's browsers support specific protocols versions only. e.g. Chrome 14, Firefox 7 and Internet Explorer 10 are currently the only browsers supporting the latest draft specification ("hybi-10") of the WebSocket protocol. The same goes for web servers. Different web servers are in varying stages of support for asynchronous messaging, with Jetty, Netty and Glassfish being the best options currently a provide native WebSocket support.

Tomcat 7 currently does not support WebSockets, yet. Check out the following issue tracker entry to learn more about the current state of affairs in Tomcat 7:

https://issues.apache.org/bugzilla/show_bug.cgi?id=51181

Socket.IO provides a default implementation for Node.JS.

It is expected that HTML5 WebSockets will replace the existing XHR approaches as well as Comet services by a new flexible and ultra high speed bidirectional TCP socket communication technology.

Technical details about WebSocket

• Uses WebSocket protocol instead of HTTP
• True full duplex communication channel; UTF8 strings and binary data can be sent in any direction at the same time.
• It is not a raw TCP socket
• Connection established by "upgrading" (handshake) from HTTP to WebSocket protocol
• Runs via port 80/443 and is firewall/proxy friendly
• Supports WebSocket ws:// and secure WebSocket wss://

Benefits of using WebSockets

• Reduces network traffic. each message has 2 bytes of overhead
• Low latency
• No polling overhead

In tests run by Kaazing Corp, who have been closely involved in the specification process, it was found that "HTML5 Web Sockets can provide a 500:1 or - depending on the size of the HTTP headers - even a 1000:1 reduction in unnecessary HTTP header traffic and 3:1 reduction in latency";.

In short: Web Sockets can make your applications faster, more efficient, and more scalable.

The WebSocket Interface:

interface WebSocket {

....

//ready state
const unsigned short CONNECTING = 0;
const unsigned short OPEN = 0;
const unsigned short CLOSING = 0;
const unsigned short CLOSED = 0;

..
//Networking
attribute Function onopen;
attribute Function onmessage;
attribute Function onerror;
attribute Function onclose;
boolean send(in data);
void close();
};

A typical Javascript client:

var wsUri = "ws://echo.websocket.org/";

function init() 
 { 
 testWebSocket(); 
 }

function testWebSocket() 
 { 
 websocket = new WebSocket(wsUri); 
 websocket.onopen = function(evt) { onOpen(evt) }; 
 websocket.onclose = function(evt) { onClose(evt) }; 
 websocket.onmessage = function(evt) { onMessage(evt) }; 
 websocket.onerror = function(evt) { onError(evt) }; 
 } 

  function onOpen(evt) 
 { 
 writeToScreen("CONNECTED"); 
 doSend("WebSocket rocks"); 
 } 

  function onClose(evt) 
 { 
 writeToScreen("DISCONNECTED"); 
 } 

  function onMessage(evt) 
 { 
 writeToScreen('RESPONSE: ' + evt.data+''); 
 websocket.close(); 
 } 

  function onError(evt) 
 { 
 writeToScreen('ERROR: ' + evt.data); 
 } 

  function doSend(message) 
 { 
 writeToScreen("SENT: " + message); 
 websocket.send(message); 
 }

function writeToScreen(message) 
 { 
 var pre = document.createElement("p"); 
 pre.innerHTML = message; 
....
 }

Getting started with WebSockets with Java backend

To build applications around websockets, I will focus on Jetty, Netty and Atmosphere for building websocket applications. The focus will be on backend processing. JQuery or raw Javascript can be used as client. We will work with a sample chat application.

1. Jetty 8

Jetty is a Java-based HTTP server and servlet container. Jetty 8 is Servlet 3.0 container and provides WebSocket implementation, so that it is possible to offer server push via both HTTP and WebSocket protocol. Jetty provides WebSocket implementation as a subclass of HttpServlet. Here is a Jetty server example:

@WebServlet(urlPatterns = "/chat", asyncSupported = true)
public class ChatServlet extends WebSocketServlet {

        // GET method is used to establish a stream connection
        @Override
        protected void doGet(HttpServletRequest request, HttpServletResponse response)
                        throws ServletException, IOException {
         //Implementation
        }

        // POST method is used to communicate with the server
        @Override
        protected void doPost(HttpServletRequest request, HttpServletResponse response)
                        throws ServletException, IOException {
  //Implementation
 }

        @Override
        public WebSocket doWebSocketConnect(HttpServletRequest request, String protocol) {
                return new ChatWebSocket();
        }

 private Queue webSockets = new ConcurrentLinkedQueue();
        class ChatWebSocket implements WebSocket.OnTextMessage {

                Connection connection;

                @Override
                public void onOpen(Connection connection) {
                        this.connection = connection;
                        webSockets.add(this);
                }

                @Override
                public void onClose(int closeCode, String message) {
                        webSockets.remove(this);
                }

                @Override
                public void onMessage(String queryString) {
                        // Parses query string
                        UrlEncoded parameters = new UrlEncoded(queryString);                        

                        Map data = new LinkedHashMap();
                        data.put("username", parameters.getString("username"));
                        data.put("message", parameters.getString("message"));


                        try {
                                messages.put(new Gson().toJson(data));
                        } catch (InterruptedException e) {
                                throw new RuntimeException(e);
                        }
                }

                @Override
                public void onClose(int closeCode, String message) {
                        webSockets.remove(this);
                }
        }
}

The advantage of this approach is that it means that WebSocket connections are terminated in the same rich application space provided by HTTP servers, thus a WebSocket enabled web application can be developed in a single environment rather than by collaboration between a HTTP server and a separate WebSocket server.

2. Atmosphere

Atmosphere is a WebSocket/Comet web framework that enables real time web application in Java. Atmosphere really simplifies a real time web application development and works with servlet containers that do not implement Servlet 3.0 but natively support Comet such as Tomcat 6. Here is an exmaple:

public class ChatAtmosphereHandler implements
                AtmosphereHandler {

        public void onRequest(AtmosphereResource resource)
                        throws IOException {
                HttpServletRequest request = resource.getRequest();
                HttpServletResponse response = resource.getResponse();

                request.setCharacterEncoding("utf-8");
                response.setCharacterEncoding("utf-8");

                // GET method is used to establish a stream connection
                if ("GET".equals(request.getMethod())) {
                        // Content-Type header
                        response.setContentType("text/plain");
                        resource.suspend();

                // POST method is used to communicate with the server
                } else if ("POST".equals(request.getMethod())) {
                        Map data = new LinkedHashMap();
                        data.put("username", request.getParameter("username"));
                        data.put("message", request.getParameter("message"));


                        // Broadcasts a message
                        resource.getBroadcaster().broadcast(new Gson().toJson(data));
                }
        }

        public void onStateChange(AtmosphereResourceEvent event)
                        throws IOException {
                if (event.getMessage() == null) {
                        return;
                }

                sendMessage(event.getResource().getResponse().getWriter(), event.getMessage().toString());
        }

        private void sendMessage(PrintWriter writer, String message) throws IOException {
                // default message format is message-size ; message-data ;
                writer.print(message.length());
                writer.print(";");
                writer.print(message);
                writer.print(";");
                writer.flush();
        }
}

Conclusion

There are many resources on www describing websockets; and many libraries trying to solve the application portability problem. To a developer trying to embrace the upcoming websockets technology, it can be confusing and overwhelming.

For others who want to integrate websockets into an existing application, there is dilemma about choosing the framework and technology stack. Building applications around HTML 5 websockets is going to be tricky for next few months till the API and protocols are standardized and the open source community provides native implementations. Using Atmosphere and jWebSockets makes sense as it abstracts out the underlying provider.

I will be writing more about Atmosphere and jWebSocket in my future blog posts.

Further reading:

http://www.html5rocks.com/en/tutorials/websockets/basics/
http://websocket.org/
http://en.wikipedia.org/wiki/WebSockets

Part 2: 

Spring annotations and Ehcache

Ehcache-spring-annotations is a library that simplifies caching in Spring based application using popular Ehcache library. In this article, I will present a simple way to integrate Ehcache in a spring based project.

Spring annotations are particularly useful when there is a need to cache methods of an application with minimal code changes and to use configuration to control the cache settings. In such cases, Ehcache Annotations can be used to dynamically configure caching of method return values.

For example, suppose you have a method: Product getProduct(long productId).

Once caching is added to this method, all calls to the method will be cached using the " productId" parameter as a key.

The steps described below works with Spring 3.1.0, Ehcache 2.5.1 and Ehcache-spring-annotations 1.2.0.

Step 1.

Configure maven to include the required libraries.

<dependency>
 <groupid>net.sf.ehcache</groupid>
 <artifactid>ehcache</artifactid>
 <version>2.5.1</version>
 <type>pom</type>
</dependency>
<dependency>
 <groupid>com.googlecode.ehcache-spring-annotations</groupid>
 <artifactid>ehcache-spring-annotations</artifactid>
 <version>1.2.0</version>
 <type>jar</type>
 <scope>compile</scope>
 <exclusions>
  <exclusion>
   <groupid>org.springframework</groupid>
   <artifactid>spring-expression</artifactid>
  </exclusion>
 </exclusions>
</dependency>
<!-- Include all spring dependencies -->
<dependency>
 <groupid>org.springframework</groupid>
 <artifactid>spring-core</artifactid>
 <version>3.1.0.RELEASE</version>
 <type>jar</type>
 <scope>compile</scope>
</dependency>

Step 2.

Configure Spring. You must add the following to your Spring configuration file in the beans declaration section:

<!-- Ehcache annotation config -->
<ehcache:annotation-driven cache-manager="ehCacheManager"/>

<bean id="ehCacheManager" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
 <property name="configLocation">
  <value>/WEB-INF/ehcache.xml</value>
 </property>
</bean>

Step 3.

Configure ehcache.xml and put it in /WEB-INF/ or in classpath.

<?xml version="1.0" encoding="UTF-8"?>
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd" updateCheck="false">

    <defaultCache eternal="false" maxElementsInMemory="1000"
        overflowToDisk="false" diskPersistent="false" timeToIdleSeconds="0"
        timeToLiveSeconds="600" memoryStoreEvictionPolicy="LRU"/>

    <cache name="product" eternal="false"
        maxElementsInMemory="100" overflowToDisk="false" diskPersistent="false"
        timeToIdleSeconds="0" timeToLiveSeconds="300"
        memoryStoreEvictionPolicy="LRU" />

</ehcache>

 If you are not familiar with configuring EhCache please read their configuration guide.

Step 4.

Add the Annotation to methods you would like to cache. Lets assume you are using the Product getProduct(long productId) method from above.

@Cacheable(cacheName = "product")
public List getProduct(long productId) {
 Query query = entityManager.createQuery(" from Product where productId = :productId");
        query.setParameter("productId", productId);
 query.setMaxResults(10);
 return query.getResultList();
}

@Cacheable annotation can be placed on a method of an interface, or a public method on a class.

Note: The cache name should match the cache name defined in ehcache.xml. Multiple cache names can be defined in ehcache.xml.

What is Spring annotations library doing in the background?

EhCacheInterceptor.java is where all the work is done. It handles invocations on methods annotated with @Cacheable. It simply calls the method and stores the value in Ehcache if the key is not already present.

//See if there is a cached result
final Element element = cache.getWithLoader(cacheKey, null, methodInvocation);
if (element != null) {
    final Object value = element.getObjectValue();

    final boolean ignoreValue = cacheInterceptor.preInvokeCachable(cache, methodInvocation, cacheKey, value);
    if (!ignoreValue) {
 return value;
    }
}
....
....
//No cached value or exception, proceed
final Object value;
try {
    value = methodInvocation.proceed();
}
if ((value != null || cacheableAttribute.isCacheNull()) && shouldCache) {
     cache.put(new Element(cacheKey, value));
}

return value;

Simple, isn't it!

Amazon's Jungle move

The rumors were in the air for a long time, as early as 2009. And it happened on a Thursday, 2nd Feb, 2012 when Junglee.com, an Amazon powered comparison shopping portal was launched. There would have been last minute glitches, pushing the release to February 2 instead of February 1.

Of the many products and services that Amazon owns, launching an aggregation and comparison portal is an interesting move. It had been tried by few Indian portals in the past but to no great success. Amazon has not done it before in any other geography. One of the reasons could be the large number of shoppers visiting Amazon.com from India. So leveraging on Junglee.com,  Amazon can build a sizable customer base for its yet to be launched marketplace.

For online shoppers, Junglee can turn out to be the starting point for majority of the transactions. Rich product description, quality product reviews and product recommendations have always helped customers make informed purchase decision.

Looking at Junglee.com, some immediate points can be made:

• A huge product selection offered on a Beta website. No API support yet.
• Of the 1.2 crore product listed, 90 lakh are books.
• The product listings are mainly from off-line sellers. Very few sellers listed have serious e-commerce presence. 

With the law on multi retail brand still hanging, setting up an E-Commerce store for Amazon is some distance away. But what will Amazon do with Junglee.com when Amazon.in is launched? 

Indeed, Amazon.com is a marketplace and they compete with other online sellers on their own platform. But in the current setup, it is unlikely that Junglee will list prominent Indian e-retailers and drive traffic to them. Of course, Junglee will drive traffic to Amazon.in apart from serving as the starting point for online shopping.

It could be a win-win situation for brick-and-mortar stores in the short term if the prices are competitive and product quality is good. Smaller cities have very few branded stores like Gitanjali, The Bombay Store and Fabindia. Shopper from these cities are already very quickly embracing online shopping.

In the past, Amazon has invested in niche shopping portals and content websites like shelfari.com, but comparison shopping portals is not really an Amazon thing. There seems to be a deeper meaning to the agenda.

Amazon has some serious competition in India. And there is a lot at stake. India is the world’s third largest e-commerce market, trailing China and the U.S. Indian online sales have doubled from around $4 billion in 2009 to nearly $10 billion in 2011, according to The Economic Times of India. Nearly $350 million has been poured into 40 Indian e-commerce start-ups as of year-end 2011 compared to $43 million in 11 companies two years prior.

There were speculations that Amazon will buy out an Indian e-retailer but it did not happen. With the amount of money some e-retailers have been throwing in advertising recently, it became a calling that the online commerce winner would be the one who brands it most on TV, print and other traditional media.

And this could be another reason for launching Junglee, to shake the tree and create some reasons to worry or even panic. Perhaps Amazon is targeting a specific competitor. True or not, there is bound to be some ripples. E-retailers with nothing new to offer will fold in. Established names will strengthen their foothold.

Whatever it is, the landscape is going to be more competitive. Customers will be spoilt for choice and price will not be the key differentiator, as is the case today.

Managing your virtual social world trail

By staying signed in on Google and performing search and accessing websites, a user leaves a digital trail behind. This trail is used by many companies to create a user profile. Richer user behavior profiles are very useful to advertisers as they can put personalized ads on websites you view.

There are some simple ways to stop this. The methods explained below will only work for advertisers who are part of Network Advertising Initiative and follows the industry privacy standards for online advertising.

Note: Before following the steps described below, ensure that you are signed in.

Google

1. Visit ads on the web. If you do not like Google 'Make the ads you see on the web more interesting', click on 'Opt Out'.

2. Visit ads preference. This is the profile Google created for you based on your search and browse history. Google even guessed your age and gender. You can delete all the information from here and 'Opt Out'.

You can also download the Advertising cookie opt-out plugin to permanently block personalized ads.

Other advertisers

1. Visit NAI. Using their online tool, you can examine your computer to identify those member companies that have placed an advertising cookie file on your computer. Don't be surprised to see 50 or more cookies from different advertisers. Now, did you know that? You can 'Select All' and 'Submit' the form to clear all the cookies.

2. Visit Aboutads. Again, you can 'opt-out' of ad network.

Facebook

1. Visit Account Settings. Click on 'Apps' tab. Delete all infrequently used apps.  For the remaining apps, set the appropriate privacy level by clicking on 'Edit'. e.g., if you use 'Washington social post reader', by default, all posts and activity from this app are visible on Facebook. Do disable this, select 'Only Me' in privacy settings.

Orkut

Orkut was once very popular in India. If you do not use it anymore and switched camps, then you are not alone. But your profile, scraps, photos etc are still public. To delete your orkut account, visit this link.

If you would like to share more privacy tips, please leave a comment.

Also read: The Internet, social media and privacy.

The Internet, Social Media and Privacy

Very recently, Google signaled its intent to begin correlating data about its users' activities across all of its most popular services and across multiple devices. The goal: to deliver those richer behavior profiles to advertisers.

Likewise, Facebook announced it will soon make Timeline - the new, glitzier user interface for its service - mandatory. Timeline is designed to chronologically assemble, automatically display and make globally accessible the preferences, acquaintances and activities for most of Facebook's 800 million members.

Combined with the addition last week of some 60 apps specifically written for Timeline, consumers can provide a detailed account, often in real time, of the music they listen to, what they eat, where they shop - even where they jog.

The driver: advertising revenue. What this tells us is that there is a lot of money at stake here. The global on-line advertising market is expected to swell to $132 billion by 2015, up from $80 billion this year, according to eMarketer. As such, it is too dangerous for 2 companies to have so much personal data. What this also tells us that there is a significant shift in the way we interact with the Internet and social networks or 'publics' in general terms, today.We are in the middle of three trends.

1. From Anonymity to Real identity

Social media has become a part of our daily lives. The things we do on social networking websites and mobile devices is increasingly about who we are.

2. From wisdom of crowd to wisdom of friends

Earlier, the Internet gave you the information in an anonymity way or the information was not personalized for you. But these days, we are more influenced by the wisdom of our friends that the wisdom of the crowd.

3. From being receivers to broadcasters

Going back in time, one had to be rich or powerful or famous in order to have a voice. But now, the power of being a broadcaster is with everyone.

How do these trends affects us? As with most changes, this freely available freedom of expression has its goods and bads.

The Internet and the social media has given us a powerful tool to speak out and be heard. Information is personalized and quickly available. Reaching out to unknown people is simple and quick. Collaborating and sharing ideas has never been so easy.

But more disquieting are the negatives. What is shocking is that some are not even aware of it. Those who are aware of it choose to silently ignore it. Some create barriers, requiring effort to understand the published information but still go on to publish. Importantly, once the information is published, regardless of our expectations, it is available and mostly remains that way. Thus, publishing personal information has become our second nature. Teenagers are most vulnerable to negative efforts of social networking.

These social networks (and malicious softwares, ISPs etc) keep track of all interactions used on their sites and save them for later use.  It is now possible to reconstruct a persons life without paying a dime or hiring a detective agency. Apps like Timeline is all you need.

A complete user profile can be created and sold to advertisers. Richer personal details are very beneficial to identity thieves and cyberspies, as well as to parties motivated to use such data unfairly against consumers, such as insurance companies, prospective employers, political campaigners and, lately, hacktivists.

With the advent of social networking website or 'publics' in general, we have become more uninhibited and often let others know more than what is really required. Mature users practice self-governance but a fair percentage of users do not. "I just checked into a restaurant!" - Well good for you, but did you ever think about a possible security threat?

“The Breakup Notifier” is another example of a Facebook “cyberstalking” app that has recently been taken down. Essentially, the application notifies users when a person breaks up with their partner through Facebook, allowing users to instantly become aware of their friend's romantic activities. Thousands had used the app within 36 hours of it's launch.

Facebook recently made sharing even easier by automatically sharing what you're doing on Facebook-connected apps. Instead of having to “Like” something to share it, you'll just need to click “Add to Timeline” on any website or app, and that app will have permission to share your activity with your Facebook friends. What activity, you may ask? It could be the news articles you read online, the videos you watch, the photos you view, the music you listen to, or any other action within the site or app. Facebook calls this auto-sharing “Gestures.” Be careful for it may cause you embarrassment.

In the web usage mining parlance, these companies are already using Clickstream for marketing (by cloaking it under the term 'relevant content') but now they are openly publishing this data for everyone to see. And thats called killing two birds with one stone.

The commonly used phrase 'Your reputation precedes you'. Knowing someone and forming opinions has become quicker.

Well, one thing is for sure and that is we will be served with relevant advertisements soon!
[For those who don't know, in September of 2003, adjacent to a New York Post article about a gruesome murder in which the victim’s body parts were stashed in a suitcase, Google listed an ad for suitcases.Since that incident, Google has improved its filters and automatically pulls ads from pages with disturbing content.]

Also read: Managing your virtual social world trail.

Suffering-Oriented Programming

While exploring the Flume architecture, I came across a presentation called 'Become Efficient or Die: The Story of BackType' that coined a new term - 'suffering-oriented programming'. It is a simple concept which means:

• Don’t add process until you feel the pain of not having it.
• Don’t build new technology until you feel the pain of not having it.
• First make it possible. Then, make it beautiful. Then, make it fast.

Examples:
Growing from 2 people to 3 people.

Other interesting points from the presentation:

Over-engineering = Attempting to create beautiful software without a thorough understanding of the problem domain.
Premature optimization = Optimizing before creating “beautiful” design, creating unnecessary complexity.
Refactoring and reducing technical debt = Garbage collection for the code base.

Technical debt:

• W needs to be refactored
• X deploy should be faster
• Y needs more unit tests
• Z needs more documentation

Such issues are never high priority to work on, but they build up and slow you down.

The presentation is available here.

The changing nature of Capitalism

We love our iPhones and iPads.

And that's why it's disconcerting to remember that the low prices of our iPhones and iPads - and the super-high profit margins of Apple - are only possible because our iPhones and iPads are made with labor practices that are that would be illegal in the United States.

The manufacturing processes of Apple and other electronics companies have come into sharp focus of late, with the revelation of details about how difficult life is for the Chinese workers who make the world's gadgets.

Here are some details:

• Foxconn, one of the companies that builds iPhones and iPads (and products for many other electronics companies), has a factory in Shenzhen that employs 430,000 people.
• According to estimates, about 5% of the workers are underage.
• The official work day in China is 8 hours long, but the standard shift is 12 hours. Generally, these shifts extend to 14-16 hours, especially when there's a hot new gadget to build. 
• The workers stay in dormitories. There are 15 beds, stacked like drawers up to the ceiling in a 12-by-12 cement cubic room.

The workers are paid ~$1 per hour or less. Manufacturing an iPhone in the United States would cost about $65 more than manufacturing it in China, where it costs an estimated $8. This additional $65 would dent the profit Apple makes on each iPhone, but it wouldn't eliminate it. (The iPhone average selling price is about $600, and Apple's average gross margin is about 40%. So Apple's gross profit on each iPhone is probably in the neighborhood of $250.)

But the reason Apple makes iPhones and iPads in China, is not just about money. The real reasons Apple makes iPhones in China are as follows:

• Most of the components of iPhones and iPads - the supply chain - are now manufactured in China, so assembling the phones half-a-world away would create huge logistical challenges. It would also reduce flexibility - the ability to switch easily from one component supplier or manufacturer to another.
• China's factories are now far bigger and more nimble than those in the United States. They can hire (and fire) tens of thousands of workers practically overnight. Because so many of the workers live on-site, they can also press them into service at a moment's notice. And they can change production practices and speeds extremely rapidly.
• China now has a far bigger supply of appropriately-qualified engineers than the U.S. does - folks with the technical skills necessary to build complex gadgets but not so credentialed that they cost too much.
• And, lastly, China's workforce is much hungrier and more frugal than many of their counterparts in the United States.

Marx made it clear that capitalism could not exist unless the worker produced a value greater than his or her own subsistence requirements. If a day's labor was required in order to keep a worker alive for a day, capital could not exist, for the day's labor would be exchanged for its own product, and capital would not be able to function as capital and consequently could not survive - If, however, a mere half-day's labor is enough to keep a worker alive during a whole day's labor, then surplus value results automatically

This surplus value does not arise in exchange, but in production. Thus the aim of production, from the capitalist's standpoint, is to get surplus value out of each worker. This is what Marx meant by the "exploitation of labor." Exploitation exists because the extra value contributed by labor is expropriated by the capitalist. Surplus value arises not because the worker is paid less than he is worth but because he produces more than he is worth.

Karl Marx was right in claiming that globalization, unfettered financial capitalism, and redistribution of income and wealth from labor to capital, could lead capitalism to self-destruct. As he argued, unregulated capitalism can lead to regular bouts of over-capacity, under-consumption and the recurrence of destructive financial crises, fueled by credit bubbles and asset-price booms and busts.

Marx argued capitalism had an internal contradiction that would cyclically lead to crises, and that contradiction - at minimum - would place intense pressure on the economic system.

The  "exploitation of labor" has effectively moved from US to China, the extent of which differs though. And it would move to another country after China. The class struggle has also begun. High unemployment and stagnant wager, according to economist Nouriel "Dr. Doom" Roubini, was triggered by the failure of laissez-faire, unregulated capitalism and free markets.

Companies are motivated to minimize costs, save money, and stockpile cash, but this leads to less money in the hands of employees, which means they have less money to spend and flow back to companies, thereby weakening the capitalist system.

The bottom line is that iPhones and iPads cost what they do because they are built using labor practices that would be illegal in this country - because people in this country consider those practices grossly unfair.

That's not a value judgment. It's a fact.

So, next time you pick up your iPhone or iPad, ask yourself how you feel about that.

Better world in 2012 and beyond...

There are reasons to believe in a better world.

A beautifully made video with moving lyrics.

It is very easy to look at the TV news and the newspapers and be quite depressed about where humanity is heading. Taking the whole advert into consideration, I think the general message is that "yes, there are many reasons to be fearful about the future, but there are even more reasons to be hopeful for the future". Hope is powerful idea – without hope, there is no reason for us to try and make amendments.

I think the overall message is best captured in one particular juxtaposition in the advert – people are still forming relationships and having children even though the media are reporting that things will get worse.

Let us start this new year with hope and goodness. Wish you all a very Happy New Year 2012!

Time series and OpenTSDB

OpenTSDB is a distributed, scalable Time Series Database (TSDB) written on top of HBase. OpenTSDB was written to address a common need: store, index and serve metrics collected from computer systems (network gear, operating systems, applications) at a large scale, and make this data easily accessible and graphable.

This article will help you gather metrics and send it to OpenTSDB server. The detailed instructions on setting up OpenTSDB can be found here.

First, register the metric.

./tsdb mkmetric order.count.1m

A simple MySQL collector.

cat > orders.sh <<\EOF
#!/bin/bash
set -e
while true; do
  echo "Querying orders...."
  mysql -u user -ppassword -Ddatabase --batch -N --execute "select count(orderdate) from ORDERS where orderdate > date_sub(now(), INTERVAL 1 MINUTE)" \
  | awk -F"\t" -v now=`date +%s` -v host=`hostname` '{ print "put order.count.1m " now " " $1 " host=" host }' \
  | nc -w 5 tsdb.host.name tsdb.host.port
  || exit  # To handle SIGPIPE properly.
  sleep 60
done
EOF

chmod +x orders.sh
nohup ./orders.sh &

This shell script will poll the MySQL database every 1 minute, get the order count for the last 1 minute and publish it to the TSDB server. Open the TSDB screen, select "order.count.1m " as metric and you should see a graph like the following:

Most of data collection work can be accomplished by writing simple shell or python scripts. Here is a code snippet that can help you publish metrics using Java.

public static void main(String[] args) throws Exception {

   Socket echoSocket = null;
   PrintWriter out = null;

   String host = null;
   int port = 0;

   while (true) {
       try {
           if (echoSocket == null) {  // Only open the connection if necessary.
                echoSocket = new Socket(host, port);
                out = new PrintWriter(echoSocket.getOutputStream(), true);
           }

           int orderCount = getOrderCount(60);//Orders in last 60 sec
           out.println("put order.count.1m " + (new Date().getTime() / 1000) + " 100");
           out.flush();
       } catch (UnknownHostException e) {
           System.err.println("Don't know about host: " + host);
           return;
       } catch (IOException e) {
           System.err.println("Couldn't get I/O for the connection to: " + host);
           out.close();
           echoSocket.close();
           out = null;
           echoSocket = null;
       }

       System.out.println("Sleeping for 60 s");
       Thread.sleep(60000);
   }
}

public int getOrderCount(int interval){
   .........
   return orderCount;
}

More collectors are available on github.

There are many good things about TSDB like:

• Simple to setup and get started.
• Easy to build dashboards. Plot and compare multiple metrics on the same graph.
• Simple HTTP API to query the time series database using "/q" option. It can produce results in HTML, JSON, PNG or plain text. This way, you can use the data for numerical analysis or use a different plotting library than the default Gnuplot used by OpenTSDB.

It will be great to see some enhancements like:

• More aggregation functions like percentiles etc.
• Better support for counters, sampling and debugging.
• User guide.
• Export to more formats like CSV, Excel etc.

StatsD is another great tool for metrics aggregation and plotting. More about it in my future blog post.

Globalization!

Shared by my friend:

My wife wanted knitting needles as aluminum needles, being relatively heavy, were proving to be painful for her. After some research, she decided on buying bamboo needles which are light weight. She could not find it online in India and so she ordered it on a UK based E-Commerce website and shipped it to her friend in London.

She was super excited after her friend arrived and handed her the needles. She quickly opened the parcel and looked at the plastic wrapper to realize the needles were "Made in India"!

Well, the needles flew all the way from India to UK and back!

The Principles!

Robustness Principle: "be conservative in what you do, be liberal in what you accept from others".

This principle encourages excellence in design and maximizes interoperability. It's practical meaning is:

• when designing software, be careful to be compliant with the protocols specified in the RFCs and STDs (Internet-related standards documents).
• do not introduce features or proprietary extensions to protocols that will be incompatible with other products or systems that are compliant.
• design your software to be tolerant of products that may not be completely compliant with Internet, Unix and Linux standards.
• where the standards are vague, accept as wide a range of reasonable operation as possible.

The principle is still honored today and is frequently mentioned in modern day Internet standards documents (RFC 2015 is just one example). Compatibility is a tradition in the world of the Internet, Unix and now Linux.

Douglas Engelbart's law of maturity: "The rate at which a person can mature is directly proportional to the embarrassment he can tolerate."

Excerpts from the book "The Engelbart Hypothesis: Dialogs with Douglas Engelbart"

"And then I’d realize, “Boy, that’s just the way I often sound.”When I was in the service I had time to think through a lot of things. I generated a sort of algorithm: the rate at which a person can mature is directly proportional to how much embarrassment he can tolerate. And I realized that embarrassment didn’t seem to bother me very much, because of my upbringing and the perspective I had about the world. Something Benjamin Franklin wrote was so beautiful, “You wouldn’t worry half so much about what other people thought about you if you realized how seldom they did,” and I’d say, “Oh, that’s right.”I seem to have a lot of intuitive capability. I just don’t mind at all not being able to explain to people how I reached something. It doesn’t bother me."

Principle of locality, also known as the principle of locality, is the phenomenon of the same value or related storage locations being frequently accessed. There are two basic types of reference locality. Temporal locality refers to the reuse of specific data and/or resources within relatively small time durations. Spatial locality refers to the use of data elements within relatively close storage locations.

New age and electronic music

New Age music is music of various styles intended to create artistic inspiration, relaxation, and optimism. It is used by listeners for yoga, massage, meditation and reading as a method of stress management or to create a peaceful atmosphere in their home or other environments, and is often associated with environmentalism and New Age spirituality.

'Beautiful' by Ryan Farish is pretty special. Listening to this album is like taking a tour through new age and electronic music. There’s some of Enigma’s arpeggio bells; a few chanting voices from Adiemus; some pleasant “whoosh” sounds from Jean Michelle Jarre; and them all topped off with some tastefully applied pan flute and simple Jim Brickman piano melodies. And yet, I think this is pretty special stuff. The compositions are nice and elegant; the melodies are simple but enriched by some intricate electronic noodling. It's really kind of intoxicating.

Enjoy 2 beautiful videos from this album.

The wifey confusion!

Friend 1: How is wify doing ?
Friend 2: do not know.. its data card...
its uses different network... something like UMTS or HDPS
Friend 1: hahahaha, wify means wife dear, not wi fi!
Friend 2: Oh!


You may also like: Thirst!, Software Humor

Interesting times in Indian E-Retailing

These are interesting times for E-Commerce in India.

Companies are getting funded by the dozens, spending money on TVCs and burning investor money, distributing gift coupons as if it were cheaper that water, and what not. In short, e-commerce is catching up!

So, an e-commerce company X, famous for selling books online, recently started offering replacement guarantee for electronic products.

Company Y, another e-commerce player in electronics space, responded with:

In these fascinating times, Taggle.com shut shop, stating competition to be too cut-throat for building a profitable business.

One e-commerce website selling electronics and gadgets since inception, entered into another category - "Toys"!

A milestone day for India’s ecommerce industry was observed recently. While most of the sites are selling mobile, tablets, perfumes etc. Ebay went a step further and sold ‘milk giving black young buffalo 20 liter per day’.

Apart from free shipping offer (national courier), the deal also comes with 7-days exchange offer!

Deal websites are not just about deal anymore. They have entered mainstream e-retailing.

Very recently, Bollywood felt left out. Yes, Bollywood. After all, who would want to miss the e-commerce money making race.

Karishma Kapoor recently invested in an e-commerce startup that operates a retail website for kids.  Ajay Devgan launched an e-ticketing and film entertainment site Ticketplease.com in January this year. Other investors in this venture include Sanjay Dutt, Bollywood producer Nitin Manmohan, Vatsal Seth, Ramakant Tibrewal, Vijay Jain and Amit Sharma of Roha JP Group. Seth has also set up Celebwears.com with actor Sohail Khan. Surely, Indian celebrities are now reaching beyond endorsements and turning into full-fledged angel investors focused on start-ups.

Finally, not all is well. dearflipkart.blogspot.com will tell you why? Interesting name for a blog, isn't it? Read the article titled Dear Flip****, Your Customer Service Has Started Sucking. Shoppers are unhappy to the extent that they have dedicated a blog and websites to voice their frustration. 

And it is just the beginning!

Thirst!

I wake up in the middle of the night, thirsty. I drink water. It doesn't quench my thirst. I drink more, and more, and more. But the thirst refuses to be quenched, like a wildfire refusing to be doused by the gallons of water dropped over it by a helicopter.

It finally dawns on me: I'm thirsty for real, but I'm drinking water in my dreams. I sigh, relieved. I decide to wake up, take pick up the bottle, and quench my thirst for real this time.

-- From Rajbir Bhattacharjee blog

You may also like: BBA entrance interview
You may also like: The wifey confusion!

NoSQL - Right tool for the right job!

First of all, the name: NoSQL is not "Never SQL" or "No To SQL". It is generally expanded to "Not only SQL".

There are a crazy number of softwares out there that are part of the 'NoSQL' movement. Some of the popular ones are listed below:

Tokyo Cabinet/Tyrant, Redis, Voldemort, Oracle BDB
CouchDB, MongoDb
Neo4J, InfoGrid, Infinite Graph
Cassandra, HBase, Riak
Oracle Coherence, db4o, ObjectStore, GemStone, Polar

NoSQL softwares can be categorized broadly into 4 categories:

1. Key-value stores
   
   • Based on Amazon's dynamo paper
   • Collection of key-value pairs
   • Example: Voldemort,  Tokyo Cabinet
2. BigTable clones
   
   • Based on Google's BigTable paper.
   • Big table, column families
   • Example: HBase, Hypertable,  Cassandra
3. Document database
   
   • Inspired by Lotus Notes
   • Collection of key-value pairs
   • Example: MongoDB, CouchDB
4. Graph Databases
   
   • Inspired by Euler and graph theory
   • Nodes, relations and key-value on both.
   • Neo4j, Sones, Allegrograph

NoSQL is primarily about scalability. Generally, when we talk about scalability, we think about the data size - from gigabytes to pentabytes and maybe more. There is another dimension to scalability - complexity of data. The two axes to scalability: Size and complexity.

Scaling size: Dealing with more and more information that is roughly similar in nature.
Scaling complexity: How do you deal with data that is more messy and more semi-structure. The categories above are mapped in the graph below.

As you can see, the focus of these softwares are very different.
Key-value stores: Simplistic data model that can massively scale.
Graph database: Capable of handling complex connected data; difficult to achieve horizontal scalability.

Typical applications:

Key-value store: Storing the customer shopping cart of a customer in an E-Commerce application. Since all lookups are based on key (customer id), and in most cases, ad-hoc query is not needed. This makes key-value an ideal storage for shopping cart data.

BigTable clones: You have a news site where any piece of content: articles, comments, author profiles, can be voted on and an optional comment supplied on the vote. You create one store per user and one store per piece of content, using a UUID as the key (generating one for each piece of content and user). The user's store holds every vote they have ever made while the content "bucket" contains a copy of every vote that has been made on the piece of content. Overnight you run a batch job to identify content that users have voted on, you generate a list of content for each user that has high votes but which they have not voted on. You then push this list of recommended articles into the user's "bucket".

Document database: Documents are semi-structured data. Modeling product attributes using RDBMS has always been a challenge. All products will have a common set of attributes e.g. title, sku, price and a category specific set of attributes. Document databases effortlessly solve this problem. MongoDB is one step away from MySQL providing ad-hoc queries, transactions, indexes and more.

Graph database: Great for cases where data is connected e.g. social applications like twitter, facebook and blogs.