Non-Essentialist:

• Avoids saying no to avoid feeling social awkwardness and pressure
• Says yes to everything

Essentialist:

• Dares to say no firmly, resolutely, and gracefully
• Says yes only to the things that really matter

Guidelines for delivering the graceful no

• Separate the decision from the relationship
• Saying no gracefully doesn’t have to mean using the word “no”
• Focus on the trade-off
• Remind yourself that everyone is selling something
• Make your peace with the fact that saying no often requires trading popularity for respect
• Remember that a clear no can be more graceful than a vague or non-committal yes

The No Repertoire

Essentialists don’t say no just occasionally. It is a part of their regular repertoire. To consistently say no with grace, it helps to have a variety of responses to call upon. Below are eight responses you can put in your no repertoire:

• The awkward pause
• The soft no (or the no but)
• Let me check my calendar and get back to you.
• Use e-mail bouncebacks.
• Say, Yes. What should I deprioritize?
• Say it with humor.
• Use the words You are welcome to X. I am willing to Y.
• I can’t do it, but X might be interested.

Credits:

These notes are from the book Essentialism: The Disciplined Pursuit of Less by Greg McKeown (Chapter #11)

Disclaimer:

These notes might be very contextual if you haven’t read the book.

Credits:

These notes are from the book Essentialism: The Disciplined Pursuit of Less by Greg McKeown (Chapter #12)

Disclaimer:

These notes might be very contextual if you haven’t read the book.

A quick guide on what (and how) to monitor to keep your workflows running smoothly.

If your organisation deals with a significant amount of data and has huge data pipelines, chances are you must have used or heard about Apache Airflow already. Airflow is an open-source workflow management platform that enables scheduling and monitoring workflows programmatically.

At Gojek, our products generate a tremendous amount of data, but that’s only step one. We’re constantly making use of that data and give value back to our customers, merchants, and partners — in the form of recommendations and other customisations. The ‘Data Engineering’ (DE) Team is responsible for building the platform and products to manage the entire lifecycle of data.

Needless to say, Airflow is one of our most heavily used tools. We cater over a thousand pipelines and an enormous amount of data using Airflow. Monitoring all these pipelines is not easy — especially considering that Airflow is still in its early phase.

Like any production application, it becomes crucial to monitor the Airflow jobs and of course, Airflow itself. It has a very resilient architecture and the design is highly scalable. It has multiple components to enable this, viz. Scheduler, Webserver, Workers, Executor, and so on. At Gojek, we have a few additional processes as well to enable flexibility for our workflows.

For example, we have a separate process running to sync our DAGs with GCS/git and a separate process to sync custom Airflow variables. We know very well that, the more components you have, higher the chances of failure. Hence, this requires a thorough monitoring and alerting system.

High-level architecture

At a high-level, we have multiple Airflow processes running in our different Kubernetes Pods, and each of them has a statsd client enabled using airflow.cfg. The statsd client will send all the metrics to Telegraf over UDP. Our custom processes are also emitting those heartbeats and other data in the same way.

We’ve configured InfluxDB as an output for Telegraf configuration (telegraf.conf) which will send the data over HTTP. You can add InfluxDB as a data source in Grafana as well as in Kapacitor. The alerts can now be configured in Kapacitor using TICK scripts which we’ll cover in the next sections.

Understanding Airflow statsd metrics

Airflow’s implementation and documentation of metrics are not the best things about it and it’s still in the early stages. In the first attempt, the measurements created by Airflow in InfluxDB were not how we wanted them to be. We solved that by writing some custom statsd telegraf templates based on the metrics name. Some of them are:

These templates will create some meaningful measurements named prefix_dag_duration, prefix_dagrun_schedule_delay, prefix_dag_loading-duration, etc in InfluxDB which can be easily queried using Grafana/Kapacitor. Following is a sample of how the fields (or tags) are parsed in InfluxDB:

The tags enable higher flexibility for querying and filtering the data. We built a Grafana dashboard on top of it as the first goal to see what’s happening under-the-hood. A sample query to generate a time series trend in Grafana for the above measurement is:

What should you monitor?

The following list contains some of the important areas that you should monitor, which could also be helpful for debugging and finding bottlenecks for resources:

• Health checks: Are scheduler, webserver, workers, and other custom processes running? What’s their uptime?
• How many workers are running?
• Are our custom metrics and configurations being reflected in metrics?
• Number of active DAGs, and DAG parsing time
• Trend: Pools usage
• Trend: Jobs execution status (started/ended)
• Trend: Executor tasks status (running/queued/open slots)
• Trend: Operator-wise execution status (failure/success)
• Trend: Task Instances status (successes/failures)
• Trend: Time taken by crucial tasks, sensors
• Trend: Time taken by the DAGs before coming to an end state
• Trend: Schedule Delay of DAGs
• Trend: Time spent by DAGs for completing dependency checks

It’s important to track these metrics at an overall level, as well as individual tasks and the DAG level. You should also consider tracking your specific operators and tasks that you think have higher chances of failure and/or consume more resources.

Alerting

Now that we have data in InfluxDB, and the monitoring is in place, we can use Kapacitor to write the TICK script to trigger alerts based on the checks and thresholds. The following snippets show some sample alerts that can be set.

// alert if scheduler is down.
stream
    |from()
        .measurement('prefix_scheduler_heartbeat')
    |deadman(0.0, 5m)
        .id('AirflowSchedulerDown')
        .message('Airflow Scheduler is down')
        .log('/tmp/alert_logs.txt')
        .trigger_alert_method()

// alert if worker count does not match the required value
stream
    |from()
        .measurement('prefix_celery_workers')
    |window()
        .period(5m)
        .every(5m)
    |alert()
        .id('Celery Workers Count')
        .crit(lambda: "value" <= 2)
        .warn(lambda: "value" <= 3)
        .log('/tmp/alert_2.txt')
        .trigger_alert_method()

Apart from this, it is also important to monitor the health of InfluxDB, Kapacitor, and Grafana as well.

References

• https://airflow.apache.org/docs/stable/metrics.html
• https://docs.influxdata.com/kapacitor/v1.5/tick/introduction/

Thank you, everyone, at the DE team — we build and manage the components described above, and special thanks to Sravan for the discussions and pairing. If you’ve questions, let us know in the comments or tweet me.

Liked what you read? Get our stories delivered straight to your inbox by signing up for Gojek newsletter. 🖖

• [430+ km] Mangalore to Panjim, Goa - 05/10/2019 with Parkar
• [60 km] Hennur Bamboo Forest - 29/09/2019 - with Parkar
• [240 km] Kolar, Avani - 240 km in 11 hour - 15/08/2019 - solo
• [130 km] Big Banayan Tree, Machanbele, Savandurga Hills - 13/01/2019 - with Parkar
• [80 km] Hesaraghatta Lake - 25/12/2017 with Parkar
• [335 km] Bangalore-Sakleshpur-Chikmangaluru - 10/2017 with Parkar
• [128 km] Nandi Hills - 17/09/2017 with Parkar
• [70 km] Mysore Trip

In this post, we’ll deploy a small Go service on Kubernetes using Terraform, locally.

Versions:

MacOS: 10.15.4
docker: 19.03.8
kubectl: client v1.15.5, server v1.14.10-gke.17
go: 1.13.9
minikube: v1.9.2
terraform: v0.12.24

You can install docker and kubectl first.

Installing Minikube

Minikube helps us run a single-node Kubernetes cluster locally. You can go through full installation details here. The prerequisite to run minikube is a hypervisor for your system. Most modern systems have virtualization enabled by default, which you can verify using this command:

# for macOS
$ sysctl -a | grep -E --color 'machdep.cpu.features|VMX'

If you see VMX highlighted in the output, it’s enabled for you. Hypervisor is the software that lets you run VM on an existing system. We’re going to use hyperkit - which is lightweight and not from Oracle 😛

# Install hyperkit
$ brew install hyperkit
# Install minikube
$ brew install minikube
# Start minikube with hyperkit
$ minikube start --vm-driver=hyperkit
# Start minikube dashboard
$ minikube dashboard

Minikube has an isolated docker environment other than the one installed by docker. To switch to that environment, use following command. All the next commands in this shell will be run on minikube docker environment.

$ eval $(minikube -p minikube docker-env)

Building a small service

We’ll now quickly build a small Go application and dockerize it.

# your directory structure will look like this
$ tree myapp
myapp
├── Dockerfile
├── Makefile
└── myapp.go

myapp.go

package main

import (
	"fmt"
	"net/http"
)

func main() {
	http.HandleFunc("/", HelloServer)
	fmt.Println("Starting server on localhost:8080")
	http.ListenAndServe(":8080", nil)
}

func HelloServer(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "Hello, %s!", r.URL.Path[1:])
}

Makefile

NAME=myapp
VERSION=0.1
all: build
build: go build -ldflags "-X main.Version=${VERSION}" myapp.go

Dockerfile

FROM golang:1.13-stretch as base
WORKDIR /build/
COPY . .
RUN ["make"]

FROM alpine:latest
WORKDIR /opt/myapp
COPY --from=base /build/myapp /opt/myapp/bin/myapp
RUN ["apk", "update"]
EXPOSE 8080

RUN ["apk", "add", "libc6-compat"]
ENTRYPOINT ["/opt/myapp/bin/myapp"]

Build and start your container using following commands (you already know these):

$ docker build -t myapp:0.1 .
$ docker run -p 8080:8080 myapp:0.1
Starting server on localhost:8080

$ curl localhost:8080
Hello, !
# our app is working fine.

Now, we’ll finally write our terraform scripts to deploy this service on minikube.

Writing the Terraform script

# here are the files that we're going to create
$ tree deploy
deploy
├── main.tf
└── outputs.tf

main.tf

provider "kubernetes" {
  config_context_cluster   = "minikube"
}

# In kubernetes, you can deploy with different type of objects
# like, Pod, ReplicationController, Deployment etc.
# While Deployment is highly recommended for production use-cases, 
# we'll simply use Pod object
resource "kubernetes_pod" "myapp" {
  metadata {
    name = "myapp"
    labels = {
      App = "myapp"
    }
  }

  spec {
    container {
      image = "myapp:0.1"
      name  = "myapp"

      port {
        container_port = 8080
      }
    }
  }
}

# Let's create a Service object - in simple words, this object helps
# you with loadbalancing and network abstraction on top of pods
resource "kubernetes_service" "myapp" {
  metadata {
    name = "myapp"
  }
  spec {
    selector = {
        App = kubernetes_pod.myapp.metadata[0].labels.App
    }
    port {
      port        = 80
      target_port = 8080
    }

    type = "NodePort" #"LoadBalancer"
  }
}

outputs.tf

output "name" {
  value = "${kubernetes_pod.myapp.metadata.0.name}"
}

Applying our Infrastructure

$ cd deploy
$ terraform init
$ terraform apply

At this point, the service should be up and running. You can verify the pods and their status. To get the IP of your service, use the following command:

$ minikube service myapp 
# this will open up your service in the brwoser window

In the next post, I’ll try to post about the minor issue I was facing in helm charts, that I was able to reproduce in local and finally fix it :)

The shoes is the picture is now finally retired, which I used for 3 years and all the Airport runs as well. Atleast 2100 km of running in these shoes overall. So far I've been successful in running a half marathon (+ 5 km) airport run on last Sunday of every year for 3 consecutive years. I plan on continuing this. I ran today because of some plans on next Sunday (actual last)

Note: The post is slightly detailed. I wanted to keep it detailed because it’s fun to read it later. This trip took place from 01-10-2019 onwards, and I started writing this post on 7th Oct, but got it finished on 26th April.

I often open up Google Maps, and search for the places/routes to visit next. India’s west coast has always been in my mind. This route consisted of many good places I always liked, plus, our Bangalore-Sakleshpur-Chikamangalur trip had the Mangalore highway in 70% of the trip; which always kept me keen to go till Mangalore.

Two weeks before the long weekend, Parkar & me discussed about this route and finalized the day-wise plan in just one sitting - a 6 days trip. Nothing else planned though. Parkar took around 5-7 warm-up rides across Bangalore during this time, and 2 days before the trip, we went for another 60 km+ warm-up ride to Hennur Forest. I was relieved after going for this warm-up ride because the sore muscles got recovered by the next day - just before the trip and, getting ready for a daily 80-100 km ride was important.

Day 0: Finding the bus to Mangalore in heavy rains (with our bikes)

We did no prior bookings (buses and hotels) in advance since we were carrying bikes and things were not fully predictable. The weather forecast showed rains all along the route till next week. We packed all our gears, dry fruits, and other edibles, and left for the Majestic bus stand. It started raining heavily just after 2 km of start (around 8 PM). It seemed like the rain is not going to stop and we didn't want to negotiate with the last bus till Mangalore; so we wore our raincoat, kept our mobile in our raincoat's pocket and resumed. It was raining heavily. Risk apart, we always liked riding in the rains. At some places, half of the tires were submerged inside the logged water, but it was fun. This was the third time I was riding in such heavy rain and so much of water.

It took us around 30-45 mins to find, convince, negotiate with a bus conductor to allow to keep our bikes in the luggage. These were KSRTC Airavat multi-axle semi-sleeper buses. We kept the bikes in vertical position supported with the bike stand inside the luggage compartment.

Inside the bus, Parkar's phone was stuck in a boot loop, and we couldn't do anything to stop it. It kept restarting the whole night. My phone was working fine, but the moment I put it in charge, it probably short-circuited and the touch sensor stopped responding. Both our phones had stopped working. We learned that raincoat pockets are not waterproof. While the bus left by 11:30 PM, I was also constantly worried about cycle getting damaged on the way, esp while crossing the ghats.

Day 1: Mangalore to Malpe Beach, Udupi, St. Mary's Island (62 km)

Getting good sleep in a semi-sleeper bus is tough, we took them because these buses have huge luggage compartments and they usually allow bikes. We got down at the Mangalore bus stand, put on our gears, and started the trip by 8 AM. This route was mostly highway - NH 66. We crossed many deltas, rivers as part of beautiful landscapes. At this point, I was slightly worried about our phones not working, but I wanted the experience to have completed this 6 days trip without phones or taking photos.

Around 1 PM, we crossed Udupi quite easily and before leaving the highway for Udupi, we found a mobile shop and enquired if mobiles could be repaired. Parkar's phone got fixed by drying out the moisture from the phone, while I had to get my screen replaced (with another broken screen because we couldn't find a new screen in that area). Once my mobile was on, the first thing I did was tranferred the house rent to my house owner in Bangalore. We asked the shopkeeper for a good place to eat - for which he suggested a local shop that cooked good sea fishes. (people are generally very helpful on the highways)

We somehow found the place he suggested and had amazing (literally) fish curry. We gorged on 3-4 types of fishes which were all mostly deep-fried. Later, we took some internal roads to reach Malpe Beach, which was pretty clean and almost empty in the afternoon. We found that boat service for St. Mary's Island is closed due to bad weather which was disappointing. So, we found a lodge nearby and took some rest. In the evening, we spent some time on the beach, watched the sunset, clicked some pictures and saw huge fish markets nearby, before having dinner at one of the shops - fish curry meal again.

Day 2: Murdeshwar (110 km)

I wouldn't say I had a very good sleep (neither did Parkar), maybe because we were not tired enough, or it wasn't home. We got up at around 6 AM, packed up, and left for Murdeshwar. Again, the route didn't disappoint us, and we crossed some beautiful landscapes which involved beaches, deltas, and ghats. Like most days, we had Idli at some local shop on the way for the breakfast.

Before coming for this trip, the weather forecast showed it'll rain for the whole week through the whole west coast. But it didn't - so far. We were having early lunch at some restaurant (Masala Dosa & coffee), and suddenly it started raining heavily. It was really bad and apparently, the people have been waiting for the rains for a couple of months. We decided we'll not wait for too long, and finally, wore the raincoat and started riding. Trust me on this, riding in the rains is indescribable - you've to experience this by yourself and especially when you're tired or on ghats.

We rode in the rains for little over one hour. We reached the beach late afternoon, searched for a place to stay and went out to eat (fish curry meal again, and some boiled eggs). Finding a good place to stay as well as to eat was not easy here since this place was slightly crowded because of the popular temple at the beach.

By 5 PM, we were at the beach for the daily sunset session.😁 We finally saw a lot of humans here, and dogs of course, since most places we went to were usually empty. At the beach, while I was feeling a bit tired for some reason, Parkar met (unusual Parkar) a large group of cyclists who were coming from Goa. They suggested a couple of internal routes that we can take, as well as some good places to stop by. I was just lying down at beach, while Parkar talked for more than an hour. By 9 PM, we left the beach and went for dinner. Every night, we would buy a liter of milk and have a protein shake before sleeping. We generally ate a lot of food to not lose a lot of weight and muscles.

Day 3: Gokarna (80 km)

Last night's sleep was not good again. For the most part of the night, I wasn't sure if I was sleeping or I was awake. It was weird. We got up early and left for Gokarna, we were comparatively more excited about reaching Gokarna. The route was a bit different now, more hilly to be specific since we were at the edge of ghats. Hilly route is a different experience - the reason is the ups and downs that we get. At one moment you're pedaling some big slopes, and suddenly there is a downward slope for some time.

Suggested by the cyclists yesterday, we decided to stop by a small footbridge over some river (Saraswati I guess) which looked good in the photos too. Since my phone was broken, Parkar was navigating the directions, but like a fool, I was cycling ahead of him. Probably because of some good song in the earphones, I was riding with very good speed and suddenly I got a call from Parkar, only to realize that I had come far ahead of the location where we had to stop. I went back and took around 20 minutes to find him (~5 km). It was a good place, and the bridge so narrow that it acted like a one-way. At the other side of the bridge, there was a broken road that seemed to go towards Gokarna, which we wanted to take, but we didn't risk our tires for it - especially tires of my road bike.

After a couple of hours, we decided to take an internal route and leave the highway (at Kumta). The good part was that this route went to Aghanashini Jetty which will directly drop us at "God's Own Beach" in Gokarna by a ferry. The bad part was that, for this odd 20 km, we had no idea about the condition of the roads. But we were excited and went for it.
This was an amazing route. Roads were good, jungles, full of trees, and hilly terrain. As we were getting close to the shore, the slope of the road was suddenly increasing. It was a sudden change, and at a few places, we even had to walk upwards. I think the roads were at 60-70 degrees angle for some time. At the peak, we were able to clearly see nearby beaches, deltas, and Gokarna! Just like how we reached on the top, there was a sudden downward slope for the next 5kms on the narrow roads. There were no vehicles on this road. The downward slope directly landed us at the ferry station, which happened very quickly. It was around 1 pm and even in the bright sun, the landscape from the ferry was mesmerizing, clouds were clear, a mix of hot and cold breezes, and we had finally touched Gokarna (my first time). The surprising part was that we just had to pay 10 bucks for the ferry ride.
Through the jungles and hilly routes in that part of Gokarna, we reached the Main beach by 3 pm. Unfortunately, everything was closed - shops, restaurants, etc and we were super hungry. We ate some peanuts and other snacks we had, found a place to stay, and took some rest. In the evening, we would spend the evening at the beach before finally having dinner at a small restaurant. Also, we parked our bikes inside the room on the third floor.😁

Day 4: Karwar, Polem, Palolem (90 km)

I finally got a very good sleep the night before. Since the route from Gokarna till Palolem was very hilly, we didn't want to end up riding in the jungles in the dark. Our goal for today was to reach Karwar and stay there. Early morning, we had to catch a ferry at Gangavali Ferry Service by 7 am - that will not just save us 15 km, but will also take us through some good routes. We were now preferring these remote routes, after our last find. We reached there on time, the huge ferry was totally empty, and before departing it was full. People used that route daily as part of their commute. Apart from saving time for everybody, it only cost 10 bucks!

After getting back to the highway, the last 30km before Karwar was very hilly, challenging and there were huge trucks passing by. These hilly routes took us through some big jungles and mountains. (edge of western ghats) By the time we reached the Karwar beach, it was only 12 pm, and even after the challenging route so far we were feeling good. So we decided that we'll continue and go till Palolem on the same day - because even in the worst case, we had 6 hours left. I clearly remember crossing the flat & straight Karwar highway behind the Karwar beach at high speed and I stopped only after reaching a flyover with an amazing view. (photo below)

After entering Goa, I wanted to stop at the Polem beach for some time which is my favorite beach. It's a remote beach and always empty. We went there but didn't waste too much time and moved ahead. The roads of South Goa are very good, have huge trees and it's hilly in most parts. Somewhere in the jungle, we spotted a restaurant. (there was nothing else in that area) We had our usual fish curry meal there, took some rest, and resumed the ride.

At one point, we saw an under construction bridge, spotted a temporary footbridge to cross Glagibag river. We went off-road and walked on that bridge to reach the Glagibagh bridge. It was rare to spot humans around so far in the trip, but a couple gave us company while crossing that bridge - who were also going to the same beach. We also realized that we were well ahead of our worst-case time scenario and Palolem is not too far. This bridge actually was part of NH-66 that took us directly to Palolem. We reached there by 4 pm, took some rest and went to the beach for the sunset, but the weather was cloudy today. Meh. At first, we thought of spending another day in Palolem, but after some discussion, we decided we'll go to Panjim next day and stay for a day.

Day 5: Agonda, Cola, Panjim (80 km)

Another good night's sleep, we were set to leave for Panjim. However, we decided to take a different route for it. Looking at the map, we saw two ways to reach Margaon. The first one was the familiar NH-66, and the other one was via Agonda and Cola - it's a state highway. At this point, we had no idea how amazing that route is going to be.

When we started riding, it was drizzling, but we kept moving. The route between Palolem and Agonda is familiar (since I had a barefoot run on this road, a year back). We were riding pretty fast, because of the amazing weather. This route was hands down the best in this whole trip. Jungles, hills, and once in a while spotting entire beaches range from the top (Agonda, Palolem etc beaches in one view). Cola was specifically amazing, this place is on ghats and within the jungle. There was nobody there (apart from a few random houses in the center of nowhere) and we could hear sounds of birds and some animals. Sometimes, it was scary as well, because we couldn't ride fast enough (or in fact we had to just walk sometimes because of the large upward slope) I was thinking, if one of those baboons attacked me, I won't even be able to run away.😂 It's hard to describe this route - drizzle, ghats, dense jungles, grass plains, birds and animals. We also spotted a French couple who were coming to Palolem from North Goa on cycle. We talked to them for a while, who were going to Kerala on a cycle over a month period.

We reached Margaon and after this, the roads were crowded with vehicles and people. By 1 pm, we had reached Panjim, and instead of looking for a room, we came to the bus stand and started searching for a bus to Bangalore. But there were no buses available - online or offline. After more than an hour of struggle and convincing about our cycles, a bus driver helped us and offered us two seats on the bus for extra money (extra money for cycles as well as for the seats). We got to know that every government bus has a couple of seats reserved for VIPs and usually it's filled at last moment. We got lucky.

The bus took the same route we came from. Within 2 hours we had already reached Karwar. From the window, we saw all the same places, restaurants and marks we had visited, and now we were passing by them in just a flash.

Afterthoughts

When we go for a long one-day trip, it requires a different mindset and preparation. Because you know that at the end you'll reach home and can rest as much as you want. But a long trip over multiple days requires a different kind of mindset. In this, you need to prepare for the next day's ride again - day after day. Both of them are an unique experience in themselves.

Additional information

Food & Nutrition
Breakfast: generally Idli, Dosa from local shops
Lunch: Fish curry meals and sometimes Dosa
Dinner: Fish curry-rice, eggs/chicken and milk + protein shake before sleeping.
We bought a huge number of water bottles throughout the trip. We always keep packets of Tang and mix in the water bottles.
Stay
We try to find lodges/hotels or home-stays between Rs. 700-1200 per night. It’s very easy to find them at highways and remote areas, plus we don’t look for anything fancy either.
Beached Visited
Malpe Beach, Maravanthe Beach, Murdeshwar Beach, Kadle Beach, Nirvana Beach, God’s Own Beach, Gokarna Main Beach, Karwar Beach, Polem Beach, Galgibaga Beach, Talpona Beach, Palolem Beach.
Bikes
Parkar: Btwin Rockride MTB
Shekhar: Btwin Triban 100
Gears
Get the list from here.

Although right now we don’t have any set plans, we’ll be back on the roads soon!

• Why this trip?
• Places & temples I visited
• Hardest part: The return trip to the home

Note: This post is slightly detailed. I wanted to keep it as a “Notes to self”; because I don’t want to forget some of the moments from this trip; especially the 3rd part of this post.

It's been a couple of months, that I had gone for a long cycling trip. 15th August on a Thursday means long weekend was coming (kind of). On Wednesday, I was thinking about the places I can go; but I and my cycling partner Nitish (we call him Parkar) had been to most Bangalore's good places in the range of around ~120 km round trip. Kolar's blue water lake (also called "Dodda" or "Chhota Ladakh") was in my mind from last few times which is roughly 52 km from Indiranagar. In the Bangalore-Sakleshpura-Chikmangaluru cycling trip (a post soon), Parkar and I had clocked our best of 135 km in a single day. I wanted to go farther than that from quite some time, but couldn't find a good destination within the range. More than that, a destination which could be safe to go; if I get tired and could crash somewhere midway.

Therefore, I decided to visit Kolar and cover some of the nearby ancient temples that I had heard of. I created a map which included the lake and 6 temples (Targetting ~220 km). Parkar couldn't join me as he was still recovering from the viral fever. In the evening, I washed the bike, did the basic checks and packed up the small decathlon bag with all the utilities (checklist here). I quickly prepared my go-to mixture of roasted grams + raisins + almonds + peanuts (~500gm overall). I knew that nutrition and hydration are going to be the deciding factor. I slept at around 1 AM since Me/Parkar/Vishwas were chilling at my home; which already delayed my plan by an hour. I got up at 6 AM.

In the morning, I had to visit Parkar's house first, to get bike's chain lube, since the bike was not serviced for a while. Then, I checked the map, plugged-in earphones, started a long playlist and set the goal to come back home by 7 PM; a total of 12.5 hours trip. I started cycling at good speed; so that while returning I could spare a few hours in case things go bad. I bought a small Indian flag 🇮🇳 on the way (NH 75) since many people were selling it on the way throughout for Independence Day celebration. At this point, I realized I had forgotten to tie Rakhi :-( It was Rakshabandhan too.

Dodda ayur Chhota Ladakh

In less than 2 hours, I had reached the Chhota Ladakh area. It was really fun so far, good songs and speed! My water bottle fell and broke due to a sudden speed breaker that I didn't see on the way. On the other hand, I couldn't find the lake for next ~30 minutes; since it was not correctly marked on the map, and no signs, etc. I moved in the wrong direction for about 2-3 km as well and then had to come back. Finally, reached the place after asking the only farmer I could see in that small village. The lake and rocks around were unexpectedly huge. The place was all empty, apart from a security guard (claimed by him); the strong winds made it a pretty chilling place.

Kolar & Avani has a very rich history. Valmiki lived here, Lav & Kush were born in Avani too. There are a large number of temples in this whole region. It’s very popular for Gold fields. (wiki)

Antharagange

I had my breakfast bars, some dry fruits and I left for Kolar city, which was around 11 km from here. This was my fastest 11 km; which I completed in 17 mins; but again the location was marked incorrectly on Google Maps. I had to ride a few extra kms here and there to finally find it. This place is full of monkeys (actual monkeys, I mean) and it has trek starting through a bunch of staircases to many caves through the rocks. It’s a Lord Shiva’s temple, which means “The Ganges from the deep” in Kannada. It’s also known as Kashi of South. (wiki) I had two tender coconuts before I left from here.

Someshwara & Kolaramma Temples

Next two temples were within the Kolar city. Most of these temples were built by Cholas time. I covered them quickly and wanted to grab a breakfast too. Google Maps was showing that Kolar had a big lake, but when I reached the location, it was just a dried out huge lake with plastics, and trees inside it. Imagine, this state of the lake was during the monsoon.

Anyway, I realized in the search of lake and a good Dosa place, I had come out of the city on a state highway (SH 96) which took me out of the city and back to NH 75. Also, I got a call from home and got scolded for not tying the rakhi yet. I borrowed a few hours from them. While cycling, you can talk to people for long. :)

I decided to have the proper breakfast after reaching my next stop, Avani Village, which was roughly 30 km from here. Plus, I have already been snacking on the items I got with me. I had a few glasses of sugarcane juice and I started for the next stop. Although, it’s not too tough to find restaurants on highways, sometimes it can get tough in remote areas; especially on a cycle.

Around 25 km later, I took a right-turn into Avani village, leaving the highway - which goes to Tirupathi. I saw a board saying it was 151 km from there, and it got me thinking if I should rather go Tirupathi, but maybe next time :) This road was better, huge trees on both sides and not wide like highways. Plus, my speed has been really good so far. I reached the next temple within 1 hour 12 mins. These villages were really small. I didn’t see a lot of people. Weather was okay, not too humid or too hot.

Ramalingeshwara Temples

These temples were a group of many temples. They were built before Chola dynasty (10th century), who had renovated these. There were barely 5-7 people around though. I roamed around, read some of the stories on the boards, and then took a nap for 15 minutes on a huge stone under a tree. I decided to leave for the next place by 1 PM. I ate most of the chikki I had, drank water and left for the next place at 1 PM.

These were really bad roads. For a stretch of over 8-10 km, I went through those patchy and broken roads. I was slightly worried about the cycle, since it has only hybrid tyres, and it’s not a mountain bike. These roads took me to some very remote areas, small villages, jungles, and very dry lands. I was constantly checking maps since there were a lot of turns and I didn’t want to take some bad route. Finally, I reached the next destination in a village.

Kotilingeshwara Temple

Unlike, all the above temples, this temple was super crowded. Suddenly, I could see a lot of humans around. This temple is not a very old temple and is also called mini-Tirupathi. It has the largest Shivling in the world and has 10 million+ small-sized Shivlings. It looked like a huge shop at first, but these Shivlings are actually donated by the devotees.

It was very crowded, and I set the time of 3 PM to start for my return trip. I had to skip the huge queue to go inside the main temple. Although they were offering free meals there, I decided to finally have my lunch outside in a local Dosa shop. Two Onion Dosas. Bought a water bottle and I left from here.

Hardest part: The return trip to the Home

I checked on the map; it was 90 km. I thought it shouldn’t take me more than 4.5 hours to reach home considering the trip so far. I’ve been feeling pretty good; except for the bad road part and was not fully tired yet.

After around next 4-5 km, I realized that I was riding at a slower speed for some reason, plus I started getting tired. Soon, I understood that I was going against the direction of winds, which was added trouble on those roads with so many up-and-downs. I was surprised to face such high impact of winds. My next goal was to cut into NH 75 near Kolar, which was around 32 km. This stretch was the toughest. I drank a lot of water, ate some more carbs, in that jungle-ish and dry land road. It was SH-96. I couldn’t ride well. But I thought, once I get back to the highway, I will be able to get the speed. So my goal was to just reach Kolar. By the way, I had not still tied the Rakhi. I could read the notifications in my watch's screen that my family's Whatsapp group was angry at me.

It took me 2.5 hours to finish this stretch. This drained out all my energy and suddenly I started to have lower back pain, knees + legs pain, and palms were getting numb. The pain was expected even before I started the trip. In all my previous trips, I never had pain in my lower legs, but this time maybe because of higher speed, it caught me. I put the pain relief spray, which eventually helped with my lower back pain after 30 minutes. But the legs were in bad shape.

After reaching Kolar on NH 75, I had 60 km to go. Unexpectedly, within 10 km, I felt I was riding really slow. The opposing wind was still there. I was worried that I might have to ride till very late in the night on this highway, which is not safe considering there are no street lights on most part of the highways. I wanted to reach the next major stop, Hoskote. which was 40 km from Kolar. I thought I was riding at a speed of 12-13 km/hour, which was really bad. Highways are un-even, there are too many ups-and-downs, flyovers, etc; which you realize only if you’re running or cycling. I had given up at this point. So I planned to take a lift till Hoskote from some truck or lorry. In my first attempt, one lorry stopped, but he said, he is going to take the next immediate left, which he actually took in 50 meters. That was slightly disappointing. I checked the maps for the distance and I realized, I was averaging exactly 20 kmph. This was good news, suddenly my calculations were on the safer side, and I felt slightly motivated. My phone battery was now 20%, so I had to turn off the music at this point. Only Google Maps was active now.

In my head, I made a plan that I’ll take a 5 minutes break, in every 30 minutes, and after every 10 km. Which means I will reach Bangalore by 7:30 PM and home before 9 PM. I broke that remaining 50 km into 5 chunks. Hoskote was 30 km from there.

My palms were aching and slightly numb. Legs were painful. But my plan was working. I was clocking exact 20 kmph. Meanwhile, It was getting dark, I kept dropping my location as checkpoints to Parkar/Vishwas in a Whatsapp group; and actually, asked them, If I don’t call them by 9 PM, give me a call. On the highway, I could see every kilometer getting reduced to Hoskote on those sideboards.

At this point, I had only the following things going in my head: 1. Tie Rakhi when you reach home. 2. Cover next 10 km in 30 minutes. 3. Reach Hoskote. 4. Eminem’s Till I Collapse verse. I couldn’t literally think of anything else happening in my life, I couldn’t sing any songs in my head either. Usually, I love the time when I ain’t thinking about anything else. eg, during a workout, or some deep work. But this was different.

Soon, I could see more trucks and street lights, and therefore I knew Hoskote was close. I filled my pocket with all the remaining roasted grams + almonds etc mix I had and kept eating it. I had reached Bangalore’s outer area, with exactly as planned time. I was very happy at this point and my next goal was to reach home :) At this point, I couldn’t speak the whole Eminem’s verse in my head either: I could only say Till I Collapse, which sometimes I even shouted to keep myself active mentally.

But Bangalore’s traffic was as usual bad in the evening. I had to stop at many signals and a few traffic jams. Problem with stopping was that it breaks the momentum and standing on the feet was more painful that pedaling the bicycle. The last 10 km turned out to be much worse, because of traffic. I reached home at exact 8:30 PM.

I drank a few sips of water and crashed on the bed. I was feeling much weaker on the bed than on the cycle; maybe because I was home. I zoned out, and got up after 30 mins, I took shower, tied all 7 Rakhis and sent the photo to my family’s Whatsapp group. Two ORS, Eat Fit’s spinach soup and half-liter milk helped me feel much better after 2 hours.

We can overcome any problem in life if we split the problem into small milestones/steps and just focus on one step at a time.

Here’s the map of the trip:

Checklist:

For Cycle:
Cycling Helmet (decathlon)
Wrench Spanner (amazon)
2 Cycle tyre tubes (decathlon)
Tyre levers (decathlon)
Multitool (decathlon)
Cycle Hand Pump (decathlon)

For you:
Cycling gloves (decathlon)
Cycle foam saddle (decathlon)
Cycling sunglasses (decathlon)
Arm UV protection cover (decathlon)
Face mask/Headband (decathlon)
Raincoat (decathlon)
Earphones

Others:
A small bag (waterproof if possible) (decathlon)
Small Plastic to cover bag or carry cash
Some cash
ID proof
Emergency contact details
Power bank
Pain relief spray
Tourch

Nutrition:
Breakfast bars (Yoga bars etc)
My favourite: Roasted Grams + Almonds + Peanuts + Raisins mix
Chikki and other dry fruits
Water bottle(s)
ORS / Tang etc.

Note: Make sure to get the items as per your cycle’s size. These links are mostly where I got them from.

Add a comment if you think I’m missing anything.

What is the MemoryError? It’s an exception thrown by interpreter when not enough memory is available for creation of new python objects or for a running operation.

The catch here is that, it doesn’t necessarily mean “not enough memory available”. It could also mean that, there are some objects that are still not cleaned up by Garbage Cleaner (GC).

To test this, I wrote a very small script:

arr = numpy.random.randn(10000000, 5)

def blast():
    for i in range(10000):
        x = pandas.DataFrame(arr.copy())
        result = x.xs(1000)

blast()

Below is the distribution (Memory usage w.r.t Time) before the program crashed with MemoryError exception.

The GC seems to be working fine, but it’s not able to clean up the objects as fast as it’s required in this case.

What’s the issue?
Python’s default implementation is CPython (github) which is implemented in C. The problem was this bug; in the implementation of malloc in glibc (which is GNU’s implementation of C standard library).

Issue Details:
M_MXFAST is the maximum size of a requested block that is served by using optimized memory containers called fastbins. free() is called when a memory cleanup of allocated space is required; which triggers the trimming of fastbins. Apparently, when malloc() is less than M_MXFAST, free() is not trimming fastbins. But, if we manually call malloc_trim(0) at that point, it should free() up those fastbins as well.

Here is a snippet from malloc.c’s free() implementation (alias __libc_free). (link)

  p = mem2chunk (mem);
  if (chunk_is_mmapped (p))                       /* release mmapped memory. */
    {
      /* See if the dynamic brk/mmap threshold needs adjusting.
         Dumped fake mmapped chunks do not affect the threshold.  */
      if (!mp_.no_dyn_threshold
          && chunksize_nomask (p) > mp_.mmap_threshold
          && chunksize_nomask (p) <= DEFAULT_MMAP_THRESHOLD_MAX
          && !DUMPED_MAIN_ARENA_CHUNK (p))
        {
          mp_.mmap_threshold = chunksize (p);
          mp_.trim_threshold = 2 * mp_.mmap_threshold;
          LIBC_PROBE (memory_mallopt_free_dyn_thresholds, 2,
                      mp_.mmap_threshold, mp_.trim_threshold);
        }
      munmap_chunk (p);
      return;
    }

Therefore, we need to trigger malloc_trim(0) from our python code written above; which we can easily do using ctypes module.

The fixed implementation looks like this:

from ctypes import cdll, CDLL
cdll.LoadLibrary("libc.so.6")
libc = CDLL("libc.so.6")
libc.malloc_trim(0)

arr = numpy.random.randn(10000000, 5)

def blast():
    for i in range(10000):
        x = pandas.DataFrame(arr.copy())
        result = x.xs(1000)
        libc.malloc_trim(0)

blast()

In another solution, I tried forcing the GC using python’s gc module; which gave the results similar to above method.

import gc

arr = numpy.random.randn(10000000, 5)

def blast():
    for i in range(10000):
        x = pandas.DataFrame(arr.copy())
        result = x.xs(1000)
        gc.collect() # Forced GC

blast()

The distrubution of Memory usage w.r.t Time looked much better now, and there was almost no difference in execution time. (see the “dark blue” line)

Similar cases, References and other notes:

1. Even after doing low_memory=False while reading a CSV using pandas.read_csv, it crashes with MemoryError exception, even though the CSV is not bigger than the RAM.
2. Explanation of malloc(), calloc(), free(), realloc() deserves a separate post altogether. I’ll post that soon.
3. Similar reported issues:
   - https://github.com/pandas-dev/pandas/issues/2659
   - https://github.com/pandas-dev/pandas/issues/21353