Yash Agarwal

Self-hosting Umami with Cloudflare Workers as Proxy

Sat, 27 Dec 2025 04:40:32 GMT

Why?

I have been using Vercel Analytics and Cloudflare Web Analytics for tracking the traffic on this blog. While both services are great, I wanted to explore self-hosting an analytics solution to have more control over my data. After some research, I decided to use Umami, a simple, open-source, privacy-focused web analytics tool. I have used Umami before, when PlanetScale used to have a free tier for their serverless MySQL database. Meanwhile, Umami also migrated from v1 to v2 and with that came the headache of database migration. Since PlanetScale had discontinued their free tier, I ditched Umami and moved to Vercel and Cloudflare for managing the analytics for this blog.

Recently, I bought a Raspberry Pi 5 to set up a home server for various projects. I thought it would be a great opportunity to self-host Umami on my Raspberry Pi. Only one issue – I don't have a UPS and I can't ensure 24x7 uptime for my Raspberry Pi server. This meant that if my home internet goes down or there's a power outage, my Umami instance would be inaccessible. I needed a solution that would allow me to self-host my Umami instance while being accessible on the public internet. I also needed a way to ensure that I don't lose any analytics data during Pi downtime.

This post outlines various steps I took to build a system matching my requirements using Cloudflare Workers as a proxy server and Cloudflare Durable Objects as temporary storage, while staying within the free tier limits of Cloudflare.

Setting up Raspberry Pi

This was the easiest part of the process. I followed the official Umami installation guide to set up Umami on my Raspberry Pi. I used Docker to run the Umami instance and connected it to a local PostgreSQL database also running in a Docker container.

This process exposed my Umami instance on my localhost at port 3000. Now I needed to find a way to access this instance from the public internet. I have past experience using Cloudflare Tunnels to expose local services to the internet so this wasn't too difficult to set up either.

Here's the Docker Compose configuration I used:

# docker-compose.yml
version: '3'
services:
  umami:
    image: ghcr.io/umami-software/umami:postgresql-latest
    ports:
      - "3000:3000"
    environment:
      DATABASE_URL: postgresql://umami:umami@db:5432/umami
      DATABASE_TYPE: postgresql
      APP_SECRET: ${APP_SECRET} # Generate with: openssl rand -base64 32
    depends_on:
      db:
        condition: service_healthy
    restart: always
 
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: umami
      POSTGRES_USER: umami
      POSTGRES_PASSWORD: umami
    volumes:
      - umami-db-data:/var/lib/postgresql/data
    restart: always
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U umami -d umami"]
      interval: 5s
      timeout: 5s
      retries: 5
 
  cloudflared:
    image: cloudflare/cloudflared:latest
    command: tunnel --no-autoupdate run
    environment:
      TUNNEL_TOKEN: ${TUNNEL_TOKEN}
    restart: always
 
volumes:
  umami-db-data:

Setting up Cloudflare Tunnel

Now comes the interesting part. I wanted to expose my Raspberry Pi directly to the public internet without a static IP or opening any ports on my router. Cloudflare Tunnels provide a secure way to expose local services to the internet without needing a public IP address or port forwarding. The tunnel creates an outbound connection from my Raspberry Pi to Cloudflare's network, allowing me to access my Umami instance securely.

Setting up the tunnel was straightforward:

Create a tunnel from the Cloudflare dashboard under Networks → Tunnels or using the Cloudflare Tunnel CLI:

cloudflared tunnel create umami-tunnel

Generate a tunnel token - The CLI autogenerates this.
Configure the tunnel token in the .env file:

# .env
TUNNEL_TOKEN=<your-tunnel-token-here>
APP_SECRET=<generate-with-openssl-rand-base64-32>

Configure ingress in the Cloudflare dashboard - this is where the magic happens. Instead of creating a public hostname like umami.yourdomain.com, I configured the ingress to route traffic from the tunnel to http://umami:3000 (the Umami service in Docker Compose).

Start everything with Docker Compose:

docker-compose up -d
 
# Verify all services are running
docker-compose ps
 
# Check cloudflared logs to confirm tunnel connection
docker-compose logs cloudflared

You should see something like:

cloudflared  | Connection <UUID> registered connIndex=0 ip=<IP> location=<LOCATION>
cloudflared  | Registered tunnel connection

At this point, my Umami instance was accessible via the private tunnel URL, but only from within Cloudflare's infrastructure. Perfect for what I needed – a secure way to access my Pi from outside my home without exposing it to the wider internet.

Setting up blog to use Umami

My first approach was simple: expose the Cloudflare Tunnel to the public internet by creating a DNS record that pointed directly to the tunnel. This allowed my blog to connect to the self-hosted Umami instance.

The next step was to configure my blog to use this Umami instance for analytics. I logged into the Umami dashboard and created a new website entry for my blog. Once configured, I copied the website ID and updated my blog's configuration to use the new Umami instance.

<head>
  <script
    is:inline
    defer
    src="https://umami.yashagarwal.in/script.js"
    data-website-id={UmamiWebsiteId}
  ></script>
</head>

With this setup, my blog was now sending analytics data to my self-hosted Umami instance running on my Raspberry Pi, accessible via the Cloudflare Tunnel.

However, what happens when my Raspberry Pi goes offline, either due to power outage or internet issues?

Without a proxy layer: requests fail when Pi is offline, resulting in lost analytics data

Since, I couldn't guarantee 24x7 uptime for my Raspberry Pi, I needed a way to ensure that analytics data was not lost during downtime. This is where Cloudflare Workers came into play.

Using Cloudflare Workers as Proxy

As clear from the above diagram, when the Raspberry Pi is offline, any requests to the Umami instance would fail, resulting in lost analytics data. To solve this problem, I decided to use Cloudflare Workers as a proxy layer between my blog and the Umami instance. Cloudflare Workers are serverless functions that run on Cloudflare's edge network, allowing me to intercept and modify requests before they reach the origin server. Since I am already using Cloudflare Tunnels to expose my Umami instance, integrating Cloudflare Workers into the setup was the most obvious choice.

To provide a storage mechanism for buffering analytics data during Pi downtime, I decided to use Cloudflare Durable Objects. It is a special type of Cloudflare Workers. Durable Objects provide a way to store stateful data in a distributed manner across Cloudflare's edge network. This meant that when the Raspberry Pi was offline, the Workers could temporarily store the analytics data in a persistent storage and forward them to the Umami instance once it came back online. Durable Objects also provide the alarm feature, which is essentially a timer that can be set to trigger a function after a specified interval. This feature is crucial for retrying the forwarding of buffered analytics data once the Raspberry Pi comes back online.

Below sequence diagram illustrates the request flow when using Cloudflare Workers as a proxy:

When a user visits my blog, the browser sends a request to the Cloudflare Workers proxy.
The Workers intercept the request and check if the Raspberry Pi is online by attempting to forward the request to the Umami instance via the Cloudflare Tunnel.
If the Raspberry Pi is online, the Workers forward the request to the Umami instance and return the response to the browser.
If the Raspberry Pi is offline, the Workers store the analytics data in a Durable Object and set an alarm to retry forwarding the data after a specified interval.

Request flow with Workers proxy: direct forwarding when Pi is online, buffering with retry when offline

Implementation Details

Cloudflare Workers Proxy

The primary task of the Cloudflare Workers proxy is to intercept requests from the blog domain, and forward them to the Umami instance running on the Raspberry Pi via the Cloudflare Tunnel. The real magic happens when the Pi is offline.

If the event forwarding to the Umami instance fails, it indicates that the Pi is offline. In this case, the Workers store the analytics data in its storage and set an alarm to retry forwarding the data after a specified interval. The client would receive a 202 Accepted response, indicating that the request has been accepted for processing, but the processing is not yet complete.

Here is a high-level pseudocode of the Workers proxy logic:

function handleEventProxy(request: Request): Response {
    // 1. Validate request
    if (request.size > MAX_SIZE) {
        return new Response('Payload Too Large', { status: 413 });
    }
 
    if (request.contentType !== 'application/json') {
        return new Response('Unsupported Media Type', { status: 415 });
    }
 
    // 2. Parse and validate payload
    const body = await request.json();
    const payload = body.payload || body;
 
    if (payload.website !== UMAMI_WEBSITE_ID) {
        return new Response('Invalid Website ID', { status: 400 });
    }
 
    // 3. Extract client metadata
    const clientIP = request.headers.get('CF-Connecting-IP');
    const userAgent = request.headers.get('User-Agent');
    const geoHeaders = extractGeolocationHeaders(request);
 
    // 4. Create event wrapper
    const event = {
        event_id: generateUUID(),
        received_at: Date.now(),
        payload: payload,
        client_ip: clientIP,
        user_agent: userAgent,
        cf_headers: geoHeaders
    };
 
    // 5. Try direct delivery
    try {
        const response = await forwardToUmami(event);
 
        if (response.ok) {
            return new Response(JSON.stringify({ success: true }), { status: 200 });
        } else {
            throw new Error(`Umami returned ${response.status}`);
        }
    } catch (error) {
        // 6. Pi is down - buffer for later
        await durableObject.buffer(event);
        return new Response(JSON.stringify({ success: true, buffered: true }), { status: 202 });
    }
}

Cloudflare Durable Object for Buffering

The Durable Object is responsible for storing the analytics data when the Raspberry Pi is offline. It maintains a queue of events and provides methods to buffer new events and retry forwarding them to the Umami instance.

When an event is buffered, the Durable Object sets an alarm to trigger a retry after a specified interval. When the alarm goes off, the Durable Object attempts to forward all buffered events to the Umami instance. If the forwarding is successful, the events are removed from the queue. If it fails again, the events remain in the queue for future retries and are retried after exponential backoff. If an event has exceeded maximum retry attempts, we stop doing exponential backoffs and retry every 30 minutes until the event has been in queue for more than 24 hours.

Here is a high-level pseudocode of the event buffering logic:

// EventBuffer Durable Object
 
async function bufferEvent(proxyEvent: ProxyEvent): Promise<Response> {
    const bufferedEvent = {
        ...proxyEvent,
        retry_count: 0,
        next_retry_at: Date.now() + 30_000, // 30s
        buffered_at: Date.now()
    };
 
    await storage.put(proxyEvent.event_id, bufferedEvent);
    ctx.waitUntil(flushBuffer());
    await ensureAlarm();
    return new Response(JSON.stringify({ buffered: true }), { status: 202 });
}
 
 
async function flushBuffer(): Promise<void> {
    const events = await storage.getAllEvents();
 
    for (const [id, event] of events) {
        if (event.next_retry_at <= Date.now()) {
            await deliverEvent(id, event);
        }
    }
 
    await ensureAlarm();
}
 
 
async function deliverEvent(id: string, event: BufferedEvent): Promise<void> {
    try {
        const response = await forwardToUmami(event);
        if (response.ok) {
            await storage.delete(id);
            return;
        }
    } catch (error) {
        // Retry logic below
    }
 
    const newRetryCount = event.retry_count + 1;
    const age = Date.now() - event.buffered_at;
 
    if (newRetryCount >= MAX_RETRIES && age > maxAge) {
        await storage.delete(id);
        return;
    }
 
    const interval = INTERVALS[newRetryCount] || MAX_INTERVAL;
    event.retry_count = newRetryCount;
    event.next_retry_at = Date.now() + interval;
    await storage.put(id, event);
}
 
 
async function ensureAlarm(): Promise<void> {
    const events = await storage.getAllEvents();
    let nextRetry = Infinity;
 
    for (const [_, event] of events) {
        if (event.next_retry_at < nextRetry) {
            nextRetry = event.next_retry_at;
        }
    }
 
    if (nextRetry !== Infinity) {
        await storage.setAlarm(nextRetry);
    } else {
        await storage.deleteAlarm();
    }
}
 
 
async function alarm(): Promise<void> {
    await flushBuffer();
}

The retry strategy follows this state flow:

Event retry flow in Durable Objects: exponential backoff with automatic cleanup after 24 hours

Conclusion

This setup evolved from my need to self-host Umami on a Raspberry Pi with high availability. I didn't want to spend money on a VPS or third-party offerings, so I leveraged Cloudflare's free tier to build a robust (for now!) system. My blog doesn't receive too much traffic, so I hope to remain within the free tier limits of Cloudflare Durable Objects for the foreseeable future. Workers with Durable Objects provide a solid fallback mechanism to ensure no analytics data is lost during Pi downtime.

I hope you enjoyed reading this post! If you have any questions or suggestions, feel free to reach out to me on X or via comments below. Do like and share if you found this post helpful!

My Bug of the Year (and How I Fixed It)

Sun, 21 Dec 2025 11:05:00 GMT

I recently worked on a Flask based microservice at work. While everything worked normally for the first few weeks, a few heavy traffic days unleashed a critical bug I had introduced inadvertently during development. This post is a detailed analysis of what went wrong and how I debugged and fixed the issue.

The Background

I have been working with Python-based distributed systems with Gevent for concurrency for the last several years. However, I never got a chance to actually write a service that uses Gevent. Most of the services I have worked with were implemented by other teams and were in production for years before I started working on them. So, I never really got to experience writing a Gevent-based Python microservice from scratch. Recently, I got an opportunity to do just that. I was tasked with writing a new microservice that was to act as an email notification service for our platform. This service was replacing another service that belonged to a different product about to reach EOL.

Service Implementation

While I initially proposed using FastAPI for the new service, the team decided to go with Flask since most of our existing services were already using Flask. So, I started working on a new Flask-based microservice that exposed REST endpoints for the internal clients. Clients would call these endpoints with the request payload. The new notification service would process the payload and send email notifications by calling another downstream service over the network within the same K8s cluster.

For the discussion in this post, the service had following tech stack -

Flask as the Web framework for implementing the business logic
Gevent-based WSGI server as the Application server to handle HTTP requests

I used Jaeger for distributed tracing so that we could trace requests across multiple services. We tested it for a couple of days in our staging environment, and everything seemed to be working fine. The service was able to handle a decent amount of load, and the traces were showing up correctly in Jaeger.

Deploying to Production

After about two weeks of testing in staging, I came up with a phase-wise rollout plan for deploying the new service to different production clusters. I started by routing a small percentage of the traffic to the new service while keeping the rest of the traffic going to the old service. I monitored the service closely for any issues. I gradually increased the traffic to the new service in phases until all the traffic was routed to the new service.

Things were looking good until this point. In fact, the service didn't show any issues for the first few weeks. However, after about a month of running in the production environment, I started noticing some anomalies in the downstream email sending service. The email service started showing an abnormally high number of incoming requests from the new notification service. Initially, I thought that the clients were simply sending more requests than before. The email service started throttling requests from the notification service due to the high load. The Grafana charts were all showing a high number of HTTP requests being processed by the notification service and an equally high number of incoming requests on the email service side. However, the logs from the clients didn't show any increase in the number of requests being sent. This was puzzling.

Symptoms start appearing

The sudden increase in the load on the email service raised quite a few eyebrows. I started investigating the issue from the notification service side to ensure that we were not dropping any notifications meant for the clients due to email service throttling. The service showed high throughput meaning the service was processing a high number of requests – an indicator of good performance in general. However the client side didn't show a significant increase in requests to the notification service. Though the number of requests from clients had increased, it was not corresponding to the number of requests being processed on the server.

Another interesting observation was that the number of context deadline exceeded errors increased on the client. The clients were written in Golang and this error generally means that the requests were timing out. In fact, the client's http client was configured to timeout after 10 seconds. Once the timeout occurs, the client retries the requests. This was an important finding.

Another thing I observed was the lack of tracing spans on the server side. I was able to find traces showing requests going out from the clients, but the spans from the Flask app layer were missing. This indicated that the Flask app was not processing requests. But then, how were the Grafana charts showing increased throughput?

From the above findings, I was able to deduce a few things -

The client throughput had definitely increased but it was not consistently high. The throughput would increase only during business hours.
The client was getting timeouts from the server and hence it was retrying with exponential backoff. The client timeout was 10 seconds per attempt, with retries continuing for up to 10 minutes total (15-17 retry attempts).
The server was showing high throughput but it was not enough to process all requests from the client within the 10s timeout window.
The downstream email service latency was around 500ms which wasn't concerning.

So, in summary, the server was processing an unusually high number of requests but was still not able to process all the client requests. The client on the other hand showed a usual rate of requests in the logs (we didn't have any metrics on client side to show the outgoing request rate at that time) and high number of timeout errors.

Finally a breakthrough!

Since I was not seeing any Jaeger spans from the Flask framework side, I suspected that the requests were not reaching the Flask server at all and were timing out (but not dropped!) at the WSGI server layer itself. When the client reaches its timeout (10s), Go cancels the context and closes the socket by calling the close() syscall. While I didn't debug the TCP layer, I suspect the client sent RST packets causing an abrupt close. When a server receives a RST packet, any reads on the socket will result in connection reset by peer and writes will result in Broken pipe error. Gevent WSGI server catches these errors, logs them and then discards the response.

Here are two possible scenarios that were in picture during this issue:

Scenario 1: WSGI server read the entire request but Flask hasn't started processing

HTTP request is already buffered in server memory
Client RST doesn't affect buffered data
When the greenlet runs, it processes the request normally
Flask executes appropriate handler
When trying to send response: BrokenPipe occurs
Gevent catches exception, logs warning, continues
Work is done, but client never knows = ZOMBIE REQUEST

Scenario 2: WSGI server read the request and Flask started processing

Client RST arrives while handler is executing
Python doesn't check socket state during processing
Handler completes normally
When trying to send response: BrokenPipe occurs
Gevent catches exception, logs warning, continues
Again a Zombie request : work done, client never notified

Since the requests already accepted by the server were not dropped even when the client sent an RST packet, every retry added more requests to the server queue. These were all duplicate requests! Flask was able to process all these queued requests eventually; along with bombarding the email service with additional duplicate requests as a side effect. That explained increased throughput on Flask and email service.

Since I was suspecting a high number of queued requests on the server side, I also decided to check the memory metrics for the Flask pods. And voila, I observed significantly increased memory usage on the pods. The CPU usage was moderate at about 50% usage consistently. But memory usage was above the 80% mark. In this process, I also found a minor bug in HPA configuration which prevented autoscaling on high memory usage. If the pods had autoscaled, the issue would not have come to light so quickly.

So, the high memory usage answered what was happening. My hypothesis was clear now - clients were sending a high number of requests, the Gevent WSGI server was accepting all these requests and spawning Greenlets to keep the requests in the memory. Flask was processing a high number of requests but it was still not enough to process all the Greenlets. Since some of the requests were timing out at the server layer, the client kept retrying the requests, adding to the load on the server.

But, why was the Flask app not able to process all the requests?

A tale of Python's single threaded nature, Gevent and monkey patching

Python is fundamentally single threaded due to GIL . Flask is also bound by this Python limitation. When a socket.recv() or requests.get() is executed, the entire process just blocks and does nothing until this I/O completes. This is particularly bad for web applications because the process can't handle more than one request at a time.

That's where Gevent shines. The solution is to use Greenlets instead of threads. Think of Greenlets as ultra-lightweight "pseudo-threads" that share the same OS thread. Instead of the operating system deciding when to switch between them ( preemptive multitasking ), Greenlets explicitly yield control to each other ( cooperative multitasking ).

Before talking about the actual fix, let's revisit our server configuration.

I had a Gevent-WSGI server and a single threaded Flask web application.
Under load, the WSGI server was able to accept multiple requests thanks to the Gevent event loop.
Every time a new request came, Gevent spawned a new greenlet.
However, since the Flask application was still single-threaded, it didn't know about Gevent's event loop or Greenlets. It still processed requests sequentially and blocked on I/O calls like requests.get.
That meant no two Greenlets were able to call Flask simultaneously.
This was evident from the memory charts (which spiked due to Greenlets in memory) and timeout errors we saw in clients (Greenlets were not able to process requests).

Here's where I messed up. I completely missed replacing Python's standard blocking I/O with Gevent-compatible versions during the development phase.

from gevent import monkey
monkey.patch_all()

The above code replaces Python's standard library at runtime . After it runs:

import socket → You get Gevent's socket (non-blocking)
import ssl → You get Gevent's ssl (yields during handshake)
time.sleep() → Yields to event loop instead of blocking
threading.Thread → Actually creates Greenlets, not threads

It's like swapping out the engine of a car while it's running. Sounds crazy, but it works.

This is what I earlier had in my main.py -

# No monkey patching here!
 
from gevent.pywsgi import WSGIServer
from flask import Flask, jsonify
import requests
 
EXTERNAL_API = "https://api.example.com/send-email"
 
app = Flask(__name__)
 
@app.route('/', methods=['POST'])
def send_email():
    payload = {}
    # Makes HTTP call to external API
    response = requests.post(EXTERNAL_API, json=payload)
    return jsonify({"status": "sent"}), 200
 
# Fire up the server
server = WSGIServer(("0.0.0.0", 9876), app)
server.serve_forever()

I was using WSGI from Gevent, so I thought I was good. Here's what actually happened when two requests arrived:

Request A arrives:

Flask starts processing
Calls requests.post() to external API
BLOCKS on socket.send() (using stdlib socket!)
Entire server frozen for 500ms

Request B arrives while A is blocked:

Connection accepted by Gevent
Request read into memory
Queued for processing (greenlet created!)
But it is stuck waiting because A's greenlet won't yield

Request processing without monkey patching: Request B waits in queue while Request A blocks the entire server

WSGI Server handles connection acceptance using Gevent's non-blocking event loop. But once the request reaches the Flask code and encounters a requests.post(), that uses the standard library socket, which blocks the entire greenlet.

Since there's only one OS thread and A's greenlet is blocked (not yielding), B's greenlet never gets a chance to run. It's like having an async server with a synchronous bottleneck.

This is the most non-trivial aspect of Gevent, and it caught me completely off-guard.

gevent.WSGIServer can accept connections without monkey patching because accept() uses Gevent's own event loop. So from the outside, it looks like your server is working. But then your application code blocks on stdlib I/O, and everything grinds to a halt.

Everything looked "normal" on the server side because the requests that did process completed in reasonable time. The problem was all the other requests timing out in the queue, which the metrics didn't capture.

How I fixed it

Adding just two lines of code fixed the issue. However, placement is critical. The monkey patching MUST happen before any other imports. Here's the proper way:

Add at the very top in main.py:

from gevent import monkey
monkey.patch_all()

The difference between "using Gevent" and "using Gevent correctly" was a single import statement that I had forgotten to add when setting up the service months earlier.

The Results

After deploying the monkey patch fix to production:

Metric	Before	After	Improvement
P95 Latency	~8-10s	~500ms	15x faster
Request Timeout Rate	~30%	< 0.1%	300x reduction
Memory Usage	85% (growing)	50% (stable)	Stabilized
Client Retry Attempts	15-17 per request	0	Eliminated
Email Service Load	700 req/min	30 req/min	Back to normal
Requests processed	2000 req/sec	50 req/sec	no retries anymore

The fix went live during a low-traffic window. Within 15 minutes, timeout errors dropped to near-zero.

All services immediately showed the significant impact of the fix. Once new pods came up, they didn't show the earlier trend of steady memory increase. Memory usage remained consistent for several days after the fix. Request latency also dropped significantly since the email service was no longer overwhelmed with duplicate requests.

Email service incoming request rate also showed a similar trend.

Lastly, the memory usage pattern of the Flask pods stabilized completely. The HPA bug I found earlier was also fixed during this deployment. The pods no longer showed increasing memory usage over time.

What I Learned

This bug taught me that understanding your framework's internals isn't optional – it's essential. I thought I was using Gevent because I imported gevent.pywsgi. I wasn't. I was using Gevent's connection handling with Python's blocking I/O, getting the worst of both worlds.

The three biggest lessons:

Read the docs thoroughly - Gevent's documentation mentions monkey patching on the first page. I skimmed it.
Question your metrics - When client and server metrics don't match, dig deeper.
Distributed tracing is your friend - Missing server spans pointed directly to the problem.

The fix was embarrassingly simple. The debugging was not. But that's software engineering – sometimes the smallest oversights create the biggest problems.

And now I have a great story for "Tell me about a time you debugged a tricky production issue" question during interviews.

Making RedBeat Work with Redis Sentinel Username Auth

Thu, 23 Oct 2025 16:05:00 GMT

A few weeks ago, while working on a new microservice at work, I encountered an interesting authentication issue with Celery's RedBeat scheduler when connecting to Redis Sentinel with username-based authentication. What seemed like a simple configuration problem turned out to be a non-trivial limitation in RedBeat's implementation. This post documents the problem, the solution, and why it matters.

The Setup

Our application stack includes:

Celery for distributed task processing
RedBeat as our dynamic Beat scheduler
Redis Sentinel for high availability
Redis ACL for username/password authentication

RedBeat is fantastic for managing dynamic schedules - you can add or modify tasks without restarting the beat process. Redis Sentinel adds high availability by automatically handling failovers when the master goes down. Both work beautifully in isolation, but combining them with Redis ACL authentication revealed an unexpected gap.

The Problem

Modern Redis deployments use ACL (Access Control Lists) that require both username and password for authentication. However, when I tried to start RedBeat with Sentinel, I kept getting authentication errors:

redis.exceptions.AuthenticationError: WRONGPASS invalid username-password pair or user is disabled.

At first, I thought it was a configuration mistake on my end. I double-checked credentials, verified Sentinel configuration, and confirmed that the username/password combination worked perfectly when connecting directly to Redis. Everything seemed correct.

Eventually, after digging through RedBeat's source code, I found the issue. When RedBeat initializes its Sentinel client in the get_redis() function, it only passes specific authentication parameters:

# From RedBeat's get_redis() function
elif conf.redis_url.startswith('redis-sentinel') and 'sentinels' in redis_options:
    connection_kwargs = {}
    if isinstance(conf.redis_use_ssl, dict):
        connection_kwargs['ssl'] = True
        connection_kwargs.update(conf.redis_use_ssl)
 
    sentinel = Sentinel(
        redis_options['sentinels'],
        socket_timeout=redis_options.get('socket_timeout'),
        password=redis_options.get('password'),  # ⚠️ Only password
        db=redis_options.get('db', 0),
        decode_responses=True,
        sentinel_kwargs=redis_options.get('sentinel_kwargs'),
        **connection_kwargs,  # SSL options spread here
    )

Notice the missing username parameter? RedBeat explicitly extracts password from redis_options and pass it to the Sentinel object, but there's no corresponding username extraction. This isn't a bug per se - it's a limitation. The underlying redis-py library fully supports username authentication, and Celery's Redis transport handles it fine. RedBeat simply doesn't expose this capability for Sentinel connections.

Why This is Non-Trivial

This problem sits at an interesting intersection:

Architectural Complexity: Redis Sentinel architecture has two types of connections - connections to Sentinel instances for service discovery, and connections to the actual Redis master for data operations. Both need authentication with ACL enabled.
No Direct Configuration: RedBeat doesn't provide a redbeat_redis_username option or similar. The standard approaches don't work:
- Can't pass username in the URL (RedBeat uses a custom Sentinel URL format)
- Can't modify sentinel_kwargs from configuration (it's hardcoded)
Security Constraints: Our production Redis deployments require ACL authentication. The default user is disabled, and password-only auth is not allowed. Downgrading security wasn't an option.

The Workaround

After studying RedBeat's code more carefully, I noticed something crucial. Look at the Sentinel initialization code again:

connection_kwargs = {}
if isinstance(conf.redis_use_ssl, dict):
    connection_kwargs['ssl'] = True
    connection_kwargs.update(conf.redis_use_ssl)  # ← Key line!
 
sentinel = Sentinel(
    redis_options['sentinels'],
    # ... other explicit parameters ...
    **connection_kwargs,  # ← This spreads our dict!
)

The redis_use_ssl configuration (which maps to redbeat_redis_use_ssl in Celery config) gets unpacked directly into the Sentinel constructor via **connection_kwargs. This happens after all the explicit parameters are set, which means anything in that dictionary gets passed through to the Sentinel client.

This is our injection point. Since the redis-py Sentinel class accepts username as a parameter, we can smuggle it through this SSL configuration dictionary. Here's how I leveraged this:

def setup_redis_sentinel_config(app_config, sentinel_nodes, auth_info):
    """
    Configure Celery and RedBeat for authenticated Sentinel connections.
 
    Args:
        app_config: Main application configuration dict
        sentinel_nodes: List of (host, port) tuples for Sentinel nodes
        auth_info: Dict containing 'username' and 'password'
    """
    service_name = "myredis-master"
 
    # Configure Celery's broker and backend
    app_config.broker_url = f"sentinel://localhost:26379"
    app_config.result_backend = f"sentinel://localhost:26379"
 
    # Celery's broker transport handles its own auth correctly
    app_config.broker_transport_options = {
        "master_name": service_name,
        "sentinels": sentinel_nodes,
        "sentinel_kwargs": {"password": None},  # Sentinels don't need auth
        "username": auth_info["username"],
        "password": auth_info["password"],
    }
 
    # RedBeat's standard config for master connections
    app_config.redbeat_redis_url = "redis-sentinel://"
    app_config.redbeat_redis_options = {
        "sentinels": sentinel_nodes,
        "service_name": service_name,
        "db": 0,
        "sentinel_kwargs": {"password": None},
        "password": auth_info["password"],
    }
 
    # THE WORKAROUND: Piggyback on redis_use_ssl to inject username
    # RedBeat unpacks this dict into connection_kwargs which gets spread
    # into the Sentinel constructor, bypassing RedBeat's parameter filtering
    app_config.redbeat_redis_use_ssl = {
        "username": auth_info["username"],
        "ssl": False,  # Critical: prevent actual SSL from being enabled
    }
 
    return app_config

The beauty of this approach is that redbeat_redis_use_ssl skips RedBeat's normal parameter validation. The dictionary contents get unpacked via **connection_kwargs directly into Sentinel(), allowing us to pass the username parameter that RedBeat's explicit parameter list ignores.

Setting ssl=False is crucial - even though RedBeat sets connection_kwargs['ssl'] = True when it detects the config, our explicit False in the spread dict takes precedence.

Conclusion

This experience demonstrates that production systems often require creative solutions when dealing with the intersection of multiple technologies. Sometimes the "right" answer isn't about writing new code, but about understanding existing code well enough to work within its constraints.

If you're facing similar issues with RedBeat and Redis Sentinel authentication, I hope this helps. And if you maintain a Celery-based application, this might be a good time to check whether your authentication setup is future-proof for ACL-based Redis deployments.

Setting Up Disqus in a Next.js Blog

Wed, 28 May 2025 11:13:24 GMT

Disqus is the most popular commenting system available in the market right now. While its free nature comes with downsides (hint - you are the data!), its ease of setup and not having to maintain a self-hosted commenting solution makes it a compelling choice for small blogs like this one.

I have been using Disqus for the last 10 years across various static site generators, so it contains all of my previous comments. While migrating Disqus comments to a different URL scheme is a topic for another day, this article focuses on setting up Disqus in a Next.js blog.

We'll cover how to:

Create a React component wrapper for Disqus
Implement lazy loading when the component is in view
Make Disqus respect theme toggling
Resolve OKLCH color incompatibility issues

Without further ado, let's begin.

Prerequisites

You should have a Next.js blog.
next-themes for handling the dark mode toggling.
disqus-react for providing Disqus script.
react-intersection-observer to implement lazy-loading.

Getting Your Disqus Shortname

Before implementing the component, you'll need to:

Create a Disqus account at disqus.com
Register your site and get your shortname
Configure your site settings in the Disqus admin panel

Install the required packages

npm install disqus-react next-themes react-intersection-observer
# OR
bun add disqus-react next-themes react-intersection-observer

Setting up next-themes

Adapt the root layout to match the following snippet -

import { ThemeProvider } from "next-themes"
 
import "./globals.css"
 
interface RootLayoutProps {
  children: React.ReactNode
}
 
export default function RootLayout({ children }: RootLayoutProps) {
  return (
    <html lang="en" suppressHydrationWarning>
      <body>
        <ThemeProvider
          attribute="class"
          defaultTheme="system"
          enableSystem
          disableTransitionOnChange
        >
          {children}
        </ThemeProvider>
      </body>
    </html>
  )
}

Now add the following to the globals.css file:

@import "tailwindcss";
 
@config "../tailwind.config.js";
 
@layer base {
  :root {
    --background: theme("colors.white");
    --foreground: theme("colors.slate.950");
  }
 
  .dark {
    --background: theme("colors.slate.950");
    --foreground: theme("colors.slate.50");
  }
 
  body {
    @apply bg-background text-foreground;
    transition:
      background-color 0.2s ease,
      color 0.2s ease;
  }
}

Key features of this setup:

suppressHydrationWarning: Prevents hydration warnings when theme changes on client
attribute="class": Uses CSS classes for theme switching (.dark class)
defaultTheme="system": Respects user's system preference
enableSystem: Allows automatic system theme detection
disableTransitionOnChange: Prevents flash during theme transitions

Since I use TailwindCSS v4, the globals.css might look a bit different. Another important thing to note is the use of syntax like theme("colors.slate.950") for setting the colors. This doesn't play nice with Disqus. We will see how to fix it later.

Disqus Component

"use client"
 
import React from "react"
import { DiscussionEmbed } from "disqus-react"
import { useTheme } from "next-themes"
import { useInView } from "react-intersection-observer"
 
interface CommentProps {
  url: string
  slug: string
}
 
export const DisqusComments = ({ url, slug }: CommentProps) => {
  const { resolvedTheme = "light" } = useTheme()
  const { ref, inView } = useInView({
    threshold: 0,
    triggerOnce: true,
    fallbackInView: true,
  })
 
  const disqusShortname = "your-disqus-shortname-here"
 
  const disqusConfig = {
    identifier: slug,
    url: url,
  }
 
  return (
    <div ref={ref}>
      {inView ? (
        <div className="comments">
          <DiscussionEmbed
            key={resolvedTheme}
            shortname={disqusShortname}
            config={disqusConfig}
          />
        </div>
      ) : null}
    </div>
  )
}

Key Features of this component are -

Lazy Loading with Intersection Observer

The component uses react-intersection-observer to implement lazy loading:
- Comments only load when the user scrolls to the comments section.
- threshold: 0 triggers loading as soon as the element enters the viewport.
- triggerOnce: true ensures the component only loads once.
- fallbackInView: true provides a fallback for environments where IntersectionObserver isn't available.
Theme-Aware Rendering

The most crucial aspect for dark mode support is the key={resolvedTheme} prop on the DiscussionEmbed component. This forces React to completely remount the Disqus embed whenever the theme changes, ensuring proper color adaptation.

Using the Disqus component

The component is used in the blog note page app/notes/[slug]/page.tsx (such as this one):

<DisqusComments
  slug={params.slug}
  url={`${siteConfig.url}/notes/${params.slug}`}
/>

The URL construction ensures each post has a unique identifier for Disqus threads, while the slug provides a fallback identifier.

Fixing OKLCH Color Incompatibility

Modern CSS frameworks like Tailwind CSS v4 use OKLCH colors for better color accuracy and perceptual uniformity. However, Disqus's embedded iframe doesn't understand OKLCH color values.

When OKLCH colors are used, you'll encounter this error in the browser console:

Uncaught Error: parseColor received unparseable color: oklch(000)

This happens because Disqus validates color values and expects them to be in RGB or Hex format.

The Solution

Override OKLCH colors with Hex equivalents specifically for the Disqus container:

/* Fix for https://github.com/disqus/disqus-react/issues/153 */
:root,
.comments {
  background-color: #ffffff !important; /* white */
  color: #0f172a !important; /* slate-950 */
}
 
/* Dark mode comments - using hex values */
:root,
.dark .comments {
  background-color: #020617 !important; /* slate-950 */
  color: #f8fafc !important; /* slate-50 */
}

Performance Considerations

Lazy Loading: Comments only load when visible, reducing initial page load time.
Theme Remounting: While remounting on theme change might seem expensive, it's necessary for proper theme switching and happens infrequently.
CSS Optimization: Color fixes are applied at the CSS level, avoiding JavaScript-based solutions that could impact performance.

Migrating from Contentlayer to Content-Collections

Sun, 25 May 2025 11:47:24 GMT

When I first moved my blog's tech stack to Next.js, Contentlayer was a rage. Obviously, I also decided to use it for managing the content here. However, over time, Contentlayer stopped getting updates, and eventually the creator declared their inability to continue its development. Keeping the package version pinned allowed me to avoid any issues for almost a year. Last year, when Next.js 15 dropped, I tried to update my blog to it, but due to Contentlayer's peer dependencies, the update failed. Finally, a few days back, I decided to put in some effort to do the migration. I had two primary choices - Contentlayer2 and content-collections. While Contentlayer2 would have been a straightforward migration, I decided to go with content-collections, simply because it seems a better tool.

The migration mostly went smoothly, apart from a few hiccups. I figured I'd document my thoughts and the steps I took, primarily as a note to myself for when I inevitably forget the implementation details.

If you are more of a code reader, then here is the PR for reference.

Why the Switch?

With Contentlayer, it was pretty straightforward to get MDX content integrated into my Next.js site. However, its development has almost stalled due to lack of funding, or at least not as active as I'd hoped in the long run.

content-collections caught my eye as a more modern and arguably more flexible alternative. It promises similar developer experience while providing a more robust and actively maintained foundation with finer-grained control over the content pipeline. I was particularly drawn to how content-collections structures its transformations and schema definitions. This approach would give me more power to shape my content exactly as needed without fighting the tooling. Though I don't currently need many of its advanced capabilities, having them available for future use is valuable.

The Migration Journey

This wasn't just a find-and-replace operation; it required understanding how my content was processed and rendered by the Contentlayer pipeline.

1. Configuration Overhaul

The biggest chunk of work was defining my content collections. Instead of the Contentlayer functions defineDocumentType and defineComputedFields, I'm now using defineCollection and defineConfig from @content-collections/core.

The filename changes too - contentlayer.config.ts -> content-collections.ts

I replicated my notes and lifelog schemas, defining the fields (title, createdOn, description, tags, etc.) using Zod for validation, which is a nice touch.

// Old (contentlayer.config.ts - conceptually)
// export const Note = defineDocumentType(() => ({ ... }))
 
// New (content-collections.ts)
const notes = defineCollection({
  name: "notes",
  directory: "content/notes",
  include: "*.mdx",
  schema: (z) => ({
    title: z.string(),
    // ... rest of the schema
  }),
  // ... transform function
})
 
export default defineConfig({
  collections: [notes, lifelog],
})

This new config file, content-collections.ts, became the single source of truth for all my content definitions and processing logic.

2. MDX Pipeline Reconstruction

This was where content-collections really showed its flexibility, and where I spent a fair bit of time re-wiring. Instead of Contentlayer's direct mdx configuration in makeSource, content-collections offers a transform function within each collection definition. This transform function provides significant flexibility.

I now explicitly call compileMDX from @content-collections/mdx within the transform function for both my notes and lifelog collections. This gave me direct access to the remark and rehype plugin chains.

// Inside transform function in content-collections.ts
const mdx = await compileMDX(context, document, {
  rehypePlugins: [
    // ... my rehype plugins like rehypeKatex, rehypePrettyCode, rehypeAutolinkHeadings
  ],
  remarkPlugins: [
    // ... my remark plugins like remarkGfm, remarkMath, remarkCallouts, remarkWikiLink
  ],
})
 
return {
  ...document,
  mdx, // The compiled MDX code is now available on `document.mdx`
  // ... other computed fields
}

This setup also allowed me to explicitly handle custom logic for readingTime and headings extraction right within the transform function, rather than relying on Contentlayer's computedFields.

Crucially, the compiled MDX content now lives under document.mdx instead of document.body.code, which meant updating all my component usage.

3. Component & Usage Updates

This was mostly a global find-and-replace, but important nonetheless:

Imports: All instances of contentlayer/generated were updated to content-collections.
MDX Rendering: My MDX component underwent a significant change.
- The getMDXComponent from next-contentlayer/hooks was replaced by useMDXComponent from @content-collections/mdx/react.
- My CustomMDXComponents (now just components internally) are passed directly to the Component rendered by useMDXComponent.
- Anywhere I was using note.body.code, I switched to note.mdx.
Data Access: In lib/content.ts, I had to adjust how I accessed certain properties. For instance, doc._meta.path became the new way to get the flattened path (what used to be slug for internal filtering), and doc.content now holds the raw markdown, replacing doc.body.raw. Even the pick utility function changed from contentlayer/client to lodash.

4. Tooling and Linting Adjustments

Follow official content-collections setup guide and migration guide for up-to-date information.

.gitignore & .prettierignore: Updated to ignore .content-collections directory.
next.config.js: Switched from withContentlayer to withContentCollections.
package.json: Installed content-collections packages and tweaked the build script (no more contentlayer build command needed explicitly, as next build now handles it through the withContentCollections plugin). This is great since I don't have to run two commands in parallel now.
tsconfig.json: Adjusted the paths alias for contentlayer/generated to content-collections.

Hiccups Along the Way

No migration is completely smooth. I also encountered one significant challenge:

1. Code Block Styling Quirks

The most noticeable friction came with rehype-pretty-code and how it interacts with the HTML output in content-collections vs. Contentlayer. Since I updated the plugin to its latest version, the code block styling completely broke down. I think the newly generated HTML structure or the way it was injected seemed to have some subtle differences.

My code blocks and inline code looked a bit off after the migration. Styles for line highlighting and character highlighting weren't applying correctly, and the general aesthetics of code blocks changed. I think this was more of a side-effect of updating the plugin rehype-pretty-code and not due to the actual migration itself.

I took this opportunity to refactor the styling a bit to make it simpler.

The selectors for rehype-pretty-code elements needed to be updated (e.g., from [data-rehype-pretty-code-fragment] to figure[data-rehype-pretty-code-figure].
I also changed the rehype-pretty-code specific class names to line--highlighted, word--highlighted.

Reflections and Takeaways

Overall, this migration was a fantastic learning experience. While the migration required several days of focused work, the benefits are tangible:

More Control: I appreciate the explicit transform function in content-collections. It feels like I have a clearer mental model of the content processing pipeline, which makes debugging and extending it much easier.
Active Development: content-collections appears to be more actively maintained and evolving, which gives me more confidence in its long-term viability for my projects.

It was a good investment of time, and I'm happy with the new content pipeline. Here's to more flexible content management and future blog articles!

Securing Traefik Docker Proxy

Sat, 17 May 2025 03:35:47 GMT

When exposing a reverse proxy like Traefik to the public internet, security should be a top priority. I recently deployed Traefik on a public-facing Linux server to route traffic to Docker containers - and I wanted a robust, layered security model to defend against port scans, brute-force attacks, and misbehaving clients.

In this post, I’ll walk through how I secured my Traefik setup using UFW, Fail2Ban, and PSAD - combining firewall rules, intrusion detection, and log-based banning into an effective security stack.

Why a Layered Security Model?

Each component of this stack plays a specific role:

UFW: Acts as our first line of defense by controlling which ports are exposed and accessible
iptables: Powers UFW and handles low-level packet filtering; we'll also configure it directly.
PSAD: Monitors network traffic to detect and block potential intrusion attempts like port scans
Fail2Ban: Analyzes application logs (Traefik in our case) to identify and block malicious behavior

Think of these tools as concentric security rings:

UFW defines what doors are open to the internet
PSAD monitors suspicious activity at those doors
Fail2Ban watches how visitors behave once they're allowed through a door

Layer 1: UFW (Uncomplicated Firewall)

UFW provides a simplified interface to iptables. My approach is "deny by default" - block everything unless explicitly allowed.

Installation

sudo apt install ufw

Configuration

# Deny all traffic by default
sudo ufw default deny outgoing comment 'deny all outgoing traffic'
sudo ufw default deny incoming comment 'deny all incoming traffic'
 
# Allow SSH (don't lock yourself out!)
sudo ufw limit in ssh comment 'allow SSH connections in'
 
# Allow web traffic for Traefik
sudo ufw allow 80/tcp comment 'allow HTTP'
sudo ufw allow 443/tcp comment 'allow HTTPS'
 
# Allow essential outbound traffic
sudo ufw allow out 53 comment 'allow DNS calls out'
sudo ufw allow out 123 comment 'allow NTP out'
sudo ufw allow out http comment 'allow HTTP traffic out'
sudo ufw allow out https comment 'allow HTTPS traffic out'

Enable and Verify

sudo ufw enable
sudo ufw status verbose

Layer 2: PSAD (Port Scan Attack Detector)

PSAD monitors network traffic through iptables logs to detect and optionally block scanning activity and other suspicious traffic.

Installation

sudo apt install psad

Configuration

Make a backup of the configuration:

sudo cp --archive /etc/psad/psad.conf /etc/psad/psad.conf-BACKUP

Edit /etc/psad/psad.conf and update these settings:

EMAIL_ADDRESSES         your@email.address;
HOSTNAME                your-server-hostname;
EXPECT_TCP_OPTIONS      Y;         # Enable TCP options logging
ENABLE_PSADWATCHD       Y;
ENABLE_AUTO_IDS         Y;         # Enable auto-blocking
ENABLE_AUTO_IDS_EMAILS  Y;         # Enable email notifications

Configure UFW to log traffic for PSAD by modifying both /etc/ufw/before.rules (for IPv4) and /etc/ufw/before6.rules (for IPv6). First make a backup of both files:

sudo cp --archive /etc/ufw/before.rules /etc/ufw/before.rules-BACKUP
sudo cp --archive /etc/ufw/before6.rules /etc/ufw/before6.rules-BACKUP

Add these lines to both files before the COMMIT line:

In /etc/ufw/before.rules:

# log all traffic so psad can analyze
 
-A ufw-before-input -j LOG --log-tcp-options --log-prefix "[IPTABLES] "
-A ufw-before-forward -j LOG --log-tcp-options --log-prefix "[IPTABLES] "

In /etc/ufw/before6.rules:

# log all traffic so psad can analyze
 
-A ufw6-before-input -j LOG --log-tcp-options --log-prefix "[IPTABLES] "
-A ufw6-before-forward -j LOG --log-tcp-options --log-prefix "[IPTABLES] "

Note: Using ufw-before-input and ufw-before-forward chains instead of directly modifying INPUT and FORWARD is crucial for proper integration with UFW.

Reload and Verify

sudo ufw reload
sudo psad -R
sudo psad --sig-update
sudo psad -H

Analyze Firewall Logs and Check PSAD Status
```
sudo psad --fw-analyze
sudo psad --Status
```
If psad --fw-analyze shows issues, you can check your mail by executing the mail command. For this to work, you need to provide <user>@localhost as the email address in EMAIL_ADDRESSES field of the psad.conf file.

Layer 3: Fail2Ban (Application Intrusion Prevention)

Fail2Ban scans log files for malicious activity and blocks offending IPs using iptables. We will configure it to monitor Traefik logs for suspicious behavior.

Installation

sudo apt install fail2ban

Basic Configuration

Create a local configuration file to avoid editing the default one:

sudo vim /etc/fail2ban/jail.local

Add the following content:

[DEFAULT]
 
# Ignore local IPs
ignoreip = 127.0.0.1/8 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12
 
# Email notifications
destemail = your@email.address
sender = your@email.address
mta = mail
 
# Get detailed email alerts
action = %(action_mwl)s

Traefik Configuration

First, make sure Traefik is configured to output access logs in a format Fail2Ban can process:

# In traefik.yml
log:
  level: debug
 
accessLog:
  filepath: /var/log/traefik/access.log
  format: common

# In docker-compose.yml
services:
  traefik:
    image: traefik:v3
    container_name: traefik
    volumes:
      - ./traefik.yml:/etc/traefik/traefik.yml:ro
      - ./logs:/var/log/traefik
      ...
      ...

Creating Traefik Filters for Fail2Ban

Create a filter for authentication failures:

sudo vim /etc/fail2ban/filter.d/traefik-auth.conf

Add this content:

[Definition]
failregex = ^<HOST> - .* "GET .* (401|403) .*$
            ^<HOST> - .* "POST .* (401|403) .*$
ignoreregex =

Create a filter for rate limiting (this will require setting up Traefik's rate limiting middleware):

sudo vim /etc/fail2ban/filter.d/traefik-ratelimit.conf

Add this content:

[Definition]
failregex = ^<HOST> - .* "GET .* (429) .*$
            ^<HOST> - .* "POST .* (429) .*$
            ^<HOST> - .* "PUT .* (429) .*$
            ^<HOST> - .* "DELETE .* (429) .*$
ignoreregex =

Setting Up Jail Configurations

Create a jail file for Traefik:

sudo vim /etc/fail2ban/jail.d/traefik.conf

Add this content:

[traefik-auth]
enabled = true
port = http,https
filter = traefik-auth
logpath = /path/to/your/traefik/logs/access.log
maxretry = 3
bantime = 1h
findtime = 10m
 
[traefik-ratelimit]
enabled = true
port = http,https
filter = traefik-ratelimit
logpath = /path/to/your/traefik/logs/access.log
maxretry = 5
bantime = 30m
findtime = 5m

Integrating Docker Networking with Security Tools

Docker's networking requires special attention to ensure our security tools work properly.

The Challenge with Docker Networks

When you use Docker, it creates its own network namespace and manipulates iptables rules to handle routing between containers and the outside world. This presents two main challenges for Fail2Ban:

Visibility Issue: In a standard Docker setup, container logs show the internal Docker network IPs (like 172.17.0.x) rather than the actual client IPs.
Ban Enforcement Issue: When Fail2Ban tries to ban an IP, it adds rules to the host's iptables, but these rules might not affect traffic that's already being handled by Docker's networking rules.

Creating a Custom Fail2Ban Action for Docker

sudo vim /etc/fail2ban/action.d/iptables-traefik-docker.conf

Add this content:

[Definition]
actionstart = iptables -N f2b-traefik
              iptables -A f2b-traefik -j RETURN
              iptables -I FORWARD -p tcp -m multiport --dports 80,443 -j f2b-traefik
 
actionstop = iptables -D FORWARD -p tcp -m multiport --dports 80,443 -j f2b-traefik
             iptables -F f2b-traefik
             iptables -X f2b-traefik
 
actionban = iptables -I f2b-traefik 1 -s <ip> -j DROP
 
actionunban = iptables -D f2b-traefik -s <ip> -j DROP

Update your jail configuration to use this action:

[traefik-auth]
# ... existing configuration
action = iptables-traefik-docker
 
[traefik-ratelimit]
# ... existing configuration
action = iptables-traefik-docker

How the Solution Works

Getting Real Client IPs in Traefik First, we need to ensure Traefik logs show the real client IPs:
- Traefik automatically captures the real client IP when properly configured
- The accesslog.format=common setting ensures these IPs are logged in a format Fail2Ban can parse
The Custom iptables Action (iptables-traefik-docker.conf) modifies iptables to work with Docker's networking:
1. On Start (actionstart):
  1. iptables -N f2b-traefik: Creates a new custom chain called "f2b-traefik"
  2. iptables -A f2b-traefik -j RETURN: Adds a default rule that returns traffic flow to the parent chain
  3. iptables -I FORWARD -p tcp -m multiport --dports 80,443 -j f2b-traefik: Inserts a rule at the beginning of the FORWARD chain to route HTTP/HTTPS traffic through our custom chain
  This setup intercepts packets in the FORWARD chain, which is crucial because Docker traffic typically passes through this chain.
2. On Ban (actionban):
  - iptables -I f2b-traefik 1 -s <ip> -j DROP: Inserts a rule at the beginning of our custom chain to drop packets from the banned IP
3. On Unban (actionunban):
  - iptables -D f2b-traefik -s <ip> -j DROP: Removes the ban rule for the specific IP
4. On Stop (actionstop):
  1. iptables -D FORWARD -p tcp -m multiport --dports 80,443 -j f2b-traefik: Removes our custom chain from the FORWARD chain
  2. iptables -F f2b-traefik: Flushes (clears) all rules in our custom chain
  3. iptables -X f2b-traefik: Deletes our custom chain

This approach ensures bans are effective even with Docker's network abstractions.

Testing the Security Setup

After configuring all components, it's important to test your setup:

Test UFW

sudo ufw status verbose

Test PSAD

sudo psad --Status

Test Fail2Ban

sudo fail2ban-client status
sudo fail2ban-client status traefik-auth
sudo fail2ban-client status traefik-ratelimit

Verify iptables Rules

sudo iptables -L f2b-traefik -n

Maintenance and Monitoring

Regular maintenance is crucial for keeping your security stack effective:

Check Logs Regularly

sudo tail -f /var/log/fail2ban.log
sudo tail -f /var/log/psad/psad.log

Update Signatures

sudo psad --sig-update

Unban IPs When Needed

sudo fail2ban-client set traefik-auth unbanip 192.168.1.100

Review Banned IPs Periodically

sudo fail2ban-client status traefik-auth

Visualizing the Attack Scenarios

To better understand how these components interact during an attack, let's visualize two common scenarios:

Port Scan Attack: An attacker scans for open ports on your server.
Brute Force Login: An attacker tries to guess login credentials for your applications.

Conclusion

Hardening a reverse proxy like Traefik on a public host isn’t a set-it-and-forget-it operation. This layered approach - combining firewall rules, intrusion detection, and log-based banning - ensures you're actively defending against real-world threats.

If you're running Docker in production or even just experimenting in a homelab, these practices are well worth the effort.

Let me know if you have any questions or if you want to share your own security setups! If you found this post helpful, consider sharing it with others who might benefit from it. Happy homelabbing!

Additional Resources

How to Secure a Linux Server

Exposing Homelab Services with Cloudflare Tunnels

Sun, 11 May 2025 07:49:47 GMT

I recently purchased a new MacMini to serve as my primary workstation. While setting up a homelab on this was a no-brainer, I wanted to ensure that I am able to access the services outside my home network. Unlike my previous attempts at homelabbing, this time, I was equipped with past experiences and a beefier machine to keep me engaged. Key objectives included enforcing HTTPS for all traffic (both local and external), and avoiding direct exposure of router ports. Since my domain is managed by Cloudflare DNS, I decided to leverage Cloudflare Tunnels for external access. This log documents the steps taken and solutions to common hiccups for future reference.

Core Design

The setup relies on two main traffic flows:

External Access:
- User navigates to service.mydomain.com.
- DNS resolves via Cloudflare, directing the request through a Cloudflare Tunnel.
- The cloudflared client (Docker container) on my workstation receives this traffic.
- cloudflared forwards the request to Traefik (another Docker container) over the internal Docker network (HTTPS).
- Traefik, acting as a reverse proxy, handles SSL termination using Let's Encrypt certificates and routes the request to the appropriate backend application container.
Internal (LAN) Access:
- User navigates to service.mydomain.com.
- My local router's custom DNS entry resolves the domain directly to my workstation's local IP.
- The browser connects directly to Traefik (listening on port 443) on the workstation.
- Traefik handles SSL and routes to the backend application.

Cloudflare Configuration - Tokens and Tunnels

Cloudflare API Token for Traefik (DNS-01 Challenge)

Traefik needs a Cloudflare API token to automatically manage Let's Encrypt certificates using Cloudflare's DNS. Here's how to create one:

Steps:

Go to Cloudflare Dashboard -> My Profile -> API Tokens.
Click "Create Token."
Choose the "Edit zone DNS" template.
Set the following Permissions: Zone - DNS - Edit
Set Zone Resources: Include - All zones from an account
Generate the token and store it securely. This will be needed in the docker-compose.yml file.

Cloudflare Tunnel Creation

To establish a secure connection from Cloudflare to my local network, I created a Cloudflare Tunnel. This tunnel acts as a secure conduit for traffic between Cloudflare's edge and my local cloudflared client.

Since I wanted to run cloudflared inside Docker on my main server, I used a separate machine to create the tunnel. This avoids needing to install cloudflared directly on my workstation.

On the separate machine:

cloudflared login - Authenticated with Cloudflare and selected mydomain.com.
cloudflared tunnel create homelab-main-tunnel - Created a new tunnel.
Noted the Tunnel ID (UUID format) provided.
cloudflared tunnel token homelab-main-tunnel - Generated the specific token for this tunnel.

Keep this token handy; it will be used in the docker-compose.yml file for the cloudflared service.

Local Project Structure and Traefik Configuration

I set up the following directory structure on my workstation:

~/homelab/
├── docker-compose.yml
├── traefik/
│   ├── traefik.yml        # Traefik static configuration
│   ├── acme.json          # For Let's Encrypt certificate storage
│   └── logs/              # Persistent logs for Traefik
└── cloudflared/
    └── config.yml         # Ingress rules for the cloudflared client

Traefik Static Configuration

This file defines Traefik's fundamental settings, entrypoints, and certificate resolvers.

global:
  checkNewVersion: true
  sendAnonymousUsage: false # Disabled for privacy
 
api:
  dashboard: true # Enable the Traefik web UI
 
log:
  level: INFO
accessLog:
  filePath: "/var/log/traefik/access.log"
 
entryPoints:
  web: # HTTP entrypoint (port 80)
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
    # docker network inspect proxy_network
    # IPAM -> Config -> Subnet
    forwardedHeaders: # Trust headers from known internal proxies
      trustedIPs:
        - "172.16.0.0/12"
        - "192.168.0.0/16"
        - "10.0.0.0/8"
 
  websecure: # HTTPS entrypoint (port 443)
    address: ":443"
    http:
      tls:
        certResolver: cloudflareResolver
        domains:
          - main: "mydomain.com"
            sans:
              - "*.mydomain.com" # Wildcard certificate for subdomains
 
providers:
  docker:
    exposedByDefault: false # Only manage containers with explicit Traefik labels
    network: proxy_network # The Docker network Traefik will monitor
 
certificatesResolvers:
  cloudflareResolver:
    acme:
      email: "my-email@example.com"
      storage: "/etc/traefik/acme.json"
      dnsChallenge:
        provider: cloudflare

Let's Encrypt Storage

Created an empty file and set permissions for Traefik to store its certificates:

touch ~/homelab/traefik/acme.json
chmod 600 ~/homelab/traefik/acme.json

Cloudflared Client Configuration

This configures the cloudflared Docker container to forward incoming tunnel traffic to Traefik's HTTPS endpoint.

# ~/homelab/cloudflared/config.yml
ingress:
  # For traffic matching my domain, forward to Traefik's HTTPS port.
  # Traefik will then handle routing based on hostname.
  - hostname: "*.mydomain.com"
    service: https://traefik:443 # 'traefik' is the Docker service name for Traefik
    originRequest:
      # Traefik's cert is for *.mydomain.com, not the internal hostname 'traefik'.
      # noTLSVerify is safe for this internal hop on a trusted Docker network.
      noTLSVerify: true
  # Default rule for any other traffic hitting the tunnel directly.
  - service: http_status:404

Rationale for https://traefik:443: I faced issues (HTTP status 418) when I tried to route to the Traefik http port. The mentioned approach routes tunnel traffic directly to Traefik's secure entrypoint. I don't know if this is the best practice, but it worked for me. The noTLSVerify option is necessary because the hostname traefik does not match the certificate's SAN (Subject Alternative Name). This is acceptable since the traffic is internal and on a trusted Docker network.

Docker Compose Orchestration

The docker-compose.yml file defines and links the Traefik, cloudflared, and application services.

version: "3.8"
 
services:
  traefik:
    image: "traefik:v3"
    container_name: "traefik_reverse_proxy"
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    env_file:
      - .env
    networks:
      - proxy_network
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro" # Allows Traefik to discover services
      - "./traefik/traefik.yml:/etc/traefik/traefik.yml:ro"
      - "./traefik/acme.json:/etc/traefik/acme.json"
      - "./traefik/logs:/var/log/traefik"
    labels:
      # Configuration for accessing the Traefik dashboard itself
      - "traefik.enable=true"
      - "traefik.http.routers.traefik-dashboard.rule=Host(`dashboard.mydomain.com`)"
      - "traefik.http.routers.traefik-dashboard.service=api@internal"
      - "traefik.http.routers.traefik-dashboard.entrypoints=websecure"
      - "traefik.http.routers.traefik-dashboard.tls.certresolver=cloudflareResolver"
      - "traefik.http.services.traefik-dashboard.loadbalancer.server.port=8080"
 
  cloudflared:
    image: "cloudflare/cloudflared:latest"
    container_name: "cloudflared_tunnel_client"
    restart: unless-stopped
    networks:
      - proxy_network
    env_file: .env
    dns:
      - 1.1.1.1
      - 1.0.0.1
    volumes:
      - "./cloudflared/config.yml:/etc/cloudflared/config.yml:ro"
    # The command uses the mounted config.yml and the tunnel token.
    command: tunnel --no-autoupdate --edge-ip-version auto --config /etc/cloudflared/config.yml run
    depends_on:
      - traefik
 
  whoami_app:
    image: "traefik/whoami"
    container_name: "whoami_service_instance"
    restart: unless-stopped
    networks:
      - proxy_network
    labels:
      - "traefik.enable=true"
      # Router definition for this application
      - "traefik.http.routers.whoami-app-router.rule=Host(`whoami.mydomain.com`)"
      - "traefik.http.routers.whoami-app-router.entrypoints=websecure"
      - "traefik.http.routers.whoami-app-router.tls.certresolver=cloudflareResolver"
      - "traefik.http.routers.whoami-app-router.service=whoami-app-service"
      # Service definition pointing to the container's port
      - "traefik.http.services.whoami-app-service.loadbalancer.server.port=80" # whoami listens on port 80
 
networks:
  proxy_network:
    name: proxy_network
    driver: bridge

DNS and Network Adjustments

At this point, the local setup is complete. However, we need to tell Cloudflare how to route traffic to our tunnel. When a request comes to the subdomain whoami.mydomain.com, it should be directed to the tunnel.

Cloudflare Public DNS Records

To add the DNS records for the tunnel, we need to create a CNAME record in Cloudflare's DNS settings. Following command works great -

cloudflared tunnel route dns homelab-main-tunnel whoami.mydomain.com

This command associates the tunnel with the subdomain whoami.mydomain.com. Cloudflare will now route traffic for this domain through the tunnel to my local cloudflared client.

Local Router DNS for Split-Brain DNS

To ensure that requests from my local network to whoami.mydomain.com do not go out to the internet and back in, I set up a split-brain DNS configuration. This allows local devices to resolve the domain directly to my workstation's local IP address. My router provides this functionality, allowing me to create custom DNS entries for specific domains.

dashboard.mydomain.com -> 192.168.X.Y (my workstation's LAN IP)
whoami.mydomain.com -> 192.168.X.Y

This ensures local traffic stays within the LAN.

Cloudflare SSL/TLS Encryption Mode

Cloudflare provides several SSL/TLS encryption modes. For my setup, I chose the "Full (Strict)" mode. This means that Cloudflare will only connect to my origin server (the cloudflared client) if it can establish a secure connection using a valid SSL certificate. This is crucial for ensuring that all traffic between Cloudflare and my local network is encrypted, even if the traffic is not going out to the public internet. I have also noticed that this mode is needed to avoid the TOO_MANY_REDIRECTS error.

In Cloudflare Dashboard -> mydomain.com -> SSL/TLS -> Overview, ensure that the SSL/TLS encryption mode is set to Full (Strict).

Hiccups and Solutions

Too many redirects

In past, I observed the error ERR_TOO_MANY_REDIRECTS almost always when Cloudflare's SSL/TLS mode was not set to "Full (Strict)". This time, apart from configuring this setting, I noticed that my docker subnet was not the standard IP range, so Traefik was not trusting X-Forwarded-Proto from cloudflared. I fixed it by setting forwardedHeaders.trustedIPs on the HTTP entrypoint to allow my docker subnet.

418 "I'm a teapot"

This was an interesting one. I am not really sure why this happened, but if I didn't set the traffic forwarding to traefik:443 in the cloudflared config, I got this error. I suspect it was because the request was not being routed correctly, and Traefik was returning a default response for unrecognized traffic.

404 Not Found

This error occurred when I tried to access the services without the correct router configuration. I had to ensure that the router for the dashboard was correctly set up in the docker-compose.yml file. Almost always, this was due to a missing or incorrect Traefik label in the service definition.

MacOS going to sleep

I noticed that my Mac mini would go to sleep after a period of inactivity, causing the tunnel to drop. To prevent this, I set the energy saver settings to "Wake for network access" This ensures that the MacMini is always listening for network connections and the tunnel stays open.

Though not applicable to my case, another important thing to note is that if you close the lid of your MacBook, it will go to sleep and the tunnel will drop. To prevent this, you can use a tool like Amphetamine or InsomniaX to keep your Mac awake while the lid is closed. Else, you can also keep the MBP plugged in to power and connected to an external display (or a virtual display adapter), which will prevent it from sleeping.

Conclusion

This setup allows me to securely access my homelab services from anywhere while keeping my local network safe. The combination of Cloudflare Tunnels, Traefik, and Docker provides a robust solution for self-hosting applications with HTTPS support. I hope this log helps others in setting up a similar configuration and serves as a reference for my future endeavors.

Running Adguard with Termux

Wed, 25 Dec 2024 09:00:00 GMT

I recently bought a new phone because after serving me faithfully for six years, my Poco F1's was left nearly unusable because of my carelessness. I dropped the phone while riding a bike and the screen cracked due to the impact. I had got the screen and battery replaced quite a few times, so I decided to finally say goodbye to this iconic phone.

For the last two months, the phone was lying unused in my closet. Recently somebody posted their homelab setup using their old phone. This post gave me the idea to bring back my Poco to life. So I started exploring the options available to me. I was planning to buy a Raspberry Pi 5, but the unit economics doesn't really work in the favour of RPi because of its rising cost. I think this phone can serve as a good alternative of RPi given that the inbuilt battery can also work as a UPS.

Specs

The Poco F1 was a beast of its time. It was launched in 2018 and came with flagship class specs.

Chipset - Qualcomm SDM845 Snapdragon 845 (10 nm)
CPU - Octa-core (4x2.8 GHz Kryo 385 Gold & 4x1.8 GHz Kryo 385 Silver)
GPU - Adreno 630
Memory - 64GB 6GB RAM

I had unlocked it in the first month of purchase and installed and used quite a few custom ROMs on it. It is still a community favorite. Alas, companies don't manufacture phones like this anymore 😔

The plan

So, given the beefy specs, I was very tempted to set something on it. There were two options available to me -

Installed something like PostMarketOS or mobile-nixos
Root the phone, install Termux and install all services with it

There is support for Linux installation on Poco, but I have some essential Google services running on this phone which can't be substituted. So I abandoned this idea.

I have previously rooted a number of phones, so rooting this phone was a trivial task. There are a number of tutorials available on the Internet if you are interested.

Initial Setup

Once the phone was rooted, I installed the Magisk module ACC and configured it to keep my battery levels between 60-75%. As I intended to keep the phone plugged into a charger 24x7, I needed to ensure that the battery is not overburdended. ACC employs certain neat tricks to ensure that battery's wear and tear is reduced while it is plugged into a charger.

Now, I wanted to set up Unbound and Adguard on this installation. I started with Adguard as it is easier to setup.

Adguard Home

Adguard Home provides arm64 binaries for arm platforms. This binary works for Android as well. So I downloaded the package from the Github releases and unpacked it. Running ./AdGuardHome/AdGuardHome under root (use pkg install tsu) worked flawlessly. This confirmed that the Adguard works on Android.

Unbound

Unbound is available on termux repositories and can be installed easily. But configuring Unbound requires some work. I have prepared the following script to help with the installation, in case you are interested -

#!/bin/bash
 
# Update package lists and install required packages
pkg update -y
pkg install -y \
    unbound \
    wget \
    openssl-tool \
    ca-certificates
 
# Create necessary directories
mkdir -p $PREFIX/etc/unbound
mkdir -p $PREFIX/var/run/unbound
 
# Download root hints file
wget https://www.internic.net/domain/named.root -O $PREFIX/etc/unbound/root.hints
 
# Generate root trust anchor for DNSSEC
# needs root to read ip_local_port_range file
su -c "/data/data/com.termux/files/usr/bin/unbound-anchor -a \"$PREFIX/etc/unbound/root.key\""
 
# Create basic configuration file
cat > $PREFIX/etc/unbound/unbound.conf << 'EOF'
server:
    # Basic server settings
    verbosity: 0
    interface: 127.0.0.1
    port: 5335
    do-ip4: yes
    do-ip6: no
    do-udp: yes
    do-tcp: yes
 
    # Access control (adjust according to your network)
    access-control: 127.0.0.0/8 allow
    access-control: 192.168.0.0/16 allow
 
    # Privacy settings
    hide-identity: yes
    hide-version: yes
 
    # Memory and performance tuning
    msg-cache-size: 50m
    rrset-cache-size: 100m
    prefetch: yes
    prefetch-key: yes
    num-threads: 2
 
    # DNSSEC
    auto-trust-anchor-file: "/data/data/com.termux/files/usr/etc/unbound/root.key"
    root-hints: "/data/data/com.termux/files/usr/etc/unbound/root.hints"
 
remote-control:
    control-enable: no
EOF
 
# Set correct permissions
chmod 644 $PREFIX/etc/unbound/unbound.conf
chmod 644 $PREFIX/etc/unbound/root.hints
chmod 644 $PREFIX/etc/unbound/root.key
 
# Test configuration
unbound-checkconf $PREFIX/etc/unbound/unbound.conf

Start services on boot

Currently, the Adguard instance is running in a terminal. As soon as the terminal is killed, the Adguard process will also stop. For a critical DNS service, this is unacceptable. So, we need to run Adguard as a standalone background service. We also need to ensure that if the device is rebooted, the services are autostarted. Termus provides a separate Termux:Boot app to allow running scripts immediately after boot. The caveat is that the user needs to unlock the phone and open termux for the Termux:boot to trigger an event which signals Termux to execute the scripts. I couldn't find a way around this limitation.

First install the Termux:Boot apk and open it at least once (as mentioned in the docs). After that you can execute the following script to set up the boot scripts. The script performs the following actions -

Create a script to start unbound on boot.
Create a script to start Adguard on boot.
Check if the services are running and create a log file.

#!/bin/bash
 
# Create directory structure
HOME_DIR=/data/data/com.termux/files/home
 
mkdir -p $HOME_DIR/.termux/boot
mkdir -p $HOME_DIR/.logs
 
# 1. Create Unbound startup script (runs first)
cat > $HOME_DIR/.termux/boot/01-unbound.sh << 'EOF'
#!/data/data/com.termux/files/usr/bin/bash
 
HOME_DIR=/data/data/com.termux/files/home
 
# Wait for system to fully boot
sleep 30
 
# Kill any existing Unbound process
su -c "pkill unbound"
 
# Start Unbound with root privileges
su -c "/data/data/com.termux/files/usr/bin/unbound -c \
    /data/data/com.termux/files/usr/etc/unbound/unbound.conf &"
 
# Log the start
echo "$(date): Unbound started" >> $HOME_DIR/.logs/unbound.log
EOF
 
# 2. Create AdGuard startup script (runs second)
cat > $HOME_DIR/.termux/boot/02-adguard.sh << 'EOF'
#!/data/data/com.termux/files/usr/bin/bash
 
HOME_DIR=/data/data/com.termux/files/home
 
# Wait for Unbound to initialize
sleep 35
 
# Kill any existing AdGuard process
su -c "pkill AdGuardHome"
 
# Start AdGuard with root privileges
su -c "$HOME_DIR/services/adguard/AdGuardHome \
    -c $HOME_DIR/services/adguard/AdGuardHome.yaml \
    -w $HOME_DIR/services/adguard \
    --no-check-update &"
 
# Log the start
echo "$(date): AdGuard started" >> $HOME_DIR/.logs/adguard.log
EOF
 
# 3. Create status check script (runs last)
cat > $HOME_DIR/.termux/boot/100-check-services.sh << 'EOF'
#!/data/data/com.termux/files/usr/bin/bash
 
HOME_DIR=/data/data/com.termux/files/home
 
# Wait for all services to start
sleep 45
 
echo "$(date): Checking services status..." >> $HOME_DIR/.logs/status.log
 
check_service() {
    if pgrep $1 > /dev/null; then
        echo "✓ $1 is running" >> $HOME_DIR/.logs/status.log
        return 0
    else
        echo "✗ $1 is NOT running" >> $HOME_DIR/.logs/status.log
        return 1
    fi
}
 
# Check each service
check_service unbound
check_service AdGuardHome
 
# Show listening ports
echo -e "\nListening ports:" >> $HOME_DIR/.logs/status.log
su -c "netstat -tupln | grep -E 'unbound|AdGuard'" >> $HOME_DIR/.logs/status.log
EOF
 
# Make all scripts executable
chmod +x $HOME_DIR/.termux/boot/*.sh

Test by rebooting your device. Once rebooted, unlock the device and open Termux and wait for 1-2 minutes. Your services should be up and running. Ensure to give forever root access to the Termux app as the services need root access to work.

Setting up Adguard

Once Adguard is up, use ifconfig to know the ip of the phone and open <ip_address>:3000 in another computer on the same subnet. The Adguard UI should show up and you can configure the settings. Below is the AdGuardHome.yaml that I used. You can place it in $HOME_DIR/services/adguard/AdGuardHome.yaml so that the boot script can use this to initialize your Adguard instance.

http:
  pprof:
    port: 6060
    enabled: false
  address: 0.0.0.0:3000
  session_ttl: 720h
users: []
auth_attempts: 5
block_auth_min: 15
http_proxy: ""
language: ""
theme: auto
dns:
  bind_hosts:
    - 0.0.0.0
  port: 53
  anonymize_client_ip: false
  ratelimit: 0
  ratelimit_subnet_len_ipv4: 24
  ratelimit_subnet_len_ipv6: 56
  ratelimit_whitelist: []
  refuse_any: true
  upstream_dns:
    - 127.0.0.1:5335
  upstream_dns_file: ""
  bootstrap_dns:
    - 9.9.9.10
    - 149.112.112.10
    - 2620:fe::10
    - 2620:fe::fe:10
  fallback_dns:
    - 1.1.1.1
  upstream_mode: load_balance
  fastest_timeout: 1s
  allowed_clients: []
  disallowed_clients: []
  blocked_hosts:
    - version.bind
    - id.server
    - hostname.bind
  trusted_proxies:
    - 127.0.0.0/8
    - ::1/128
  cache_size: 0
  cache_ttl_min: 0
  cache_ttl_max: 0
  cache_optimistic: false
  bogus_nxdomain: []
  aaaa_disabled: false
  enable_dnssec: false
  edns_client_subnet:
    custom_ip: ""
    enabled: false
    use_custom: false
  max_goroutines: 300
  handle_ddr: true
  ipset: []
  ipset_file: ""
  bootstrap_prefer_ipv6: false
  upstream_timeout: 10s
  private_networks: []
  use_private_ptr_resolvers: true
  local_ptr_upstreams:
    - 192.168.1.1
  use_dns64: false
  dns64_prefixes: []
  serve_http3: false
  use_http3_upstreams: false
  serve_plain_dns: true
  hostsfile_enabled: true
tls:
  enabled: false
  server_name: ""
  force_https: false
  port_https: 443
  port_dns_over_tls: 853
  port_dns_over_quic: 853
  port_dnscrypt: 0
  dnscrypt_config_file: ""
  allow_unencrypted_doh: false
  certificate_chain: ""
  private_key: ""
  certificate_path: ""
  private_key_path: ""
  strict_sni_check: false
querylog:
  dir_path: ""
  ignored: []
  interval: 2160h
  size_memory: 1000
  enabled: true
  file_enabled: true
statistics:
  dir_path: ""
  ignored: []
  interval: 2160h
  enabled: true
filters:
  - enabled: false
    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt
    name: AdGuard DNS filter
    id: 1
  - enabled: false
    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_2.txt
    name: AdAway Default Blocklist
    id: 2
  - enabled: true
    url: https://adguardteam.github.io/HostlistsRegistry/assets/filter_51.txt
    name: HaGeZi's Pro++ Blocklist
    id: 1735062048
whitelist_filters: []
user_rules: []
dhcp:
  enabled: false
  interface_name: ""
  local_domain_name: lan
  dhcpv4:
    gateway_ip: ""
    subnet_mask: ""
    range_start: ""
    range_end: ""
    lease_duration: 86400
    icmp_timeout_msec: 1000
    options: []
  dhcpv6:
    range_start: ""
    lease_duration: 86400
    ra_slaac_only: false
    ra_allow_slaac: false
filtering:
  blocking_ipv4: ""
  blocking_ipv6: ""
  blocked_services:
    schedule:
      time_zone: UTC
    ids: []
  protection_disabled_until: null
  safe_search:
    enabled: false
    bing: true
    duckduckgo: true
    ecosia: true
    google: true
    pixabay: true
    yandex: true
    youtube: true
  blocking_mode: default
  parental_block_host: family-block.dns.adguard.com
  safebrowsing_block_host: standard-block.dns.adguard.com
  rewrites: []
  safe_fs_patterns: []
  safebrowsing_cache_size: 1048576
  safesearch_cache_size: 1048576
  parental_cache_size: 1048576
  cache_time: 30
  filters_update_interval: 0
  blocked_response_ttl: 10
  filtering_enabled: true
  parental_enabled: false
  safebrowsing_enabled: false
log:
  enabled: true
  file: ""
  max_backups: 0
  max_size: 100
  max_age: 3
  compress: false
  local_time: false
  verbose: false
os:
  group: ""
  user: ""
  rlimit_nofile: 0
schema_version: 29

Please make sure that you modify this configuration according to your network setup. For example, you may want to use your router's address in the local_ptr_upstreams field.

That brings us to the end of this post. Please keep sharing it if you find it useful.

Debug Bluetooth on Linux

Fri, 17 May 2024 15:55:15 GMT

Recently, I installed Debian minimal on my old laptop. The earlier plan was to use this as a server, but I changed my mind and settled on using it as a development server, to avoid storing personal code on my work laptop. The intention was to use it as a headless machine and keep it connected to a monitor and keyboard-mouse pair to interact with it. I was able to connect my monitor to this laptop (after setting up xrandr) but the bluetooth on the machine failed to turn on. This post details various steps I took to root cause and fix the issue.

Install the basic packages

To make bluetooth work on Debian/Ubuntu systems, following packages should be installed -

sudo apt install bluetooth pulseaudio-module-bluetooth blueman bluez-firmware bluez-tools

Now enable the bluetooth service -

sudo systemctl enable bluetooth.service
sudo systemctl start bluetooth.service

You can also enable the kernel bluetooth module using this command -

modprobe btusb

Now, restart the system (it is optional, but recommended).

Reset bluetooth adapter

Use following command to reset bluetooth adapter -

sudo hciconfig hci0 reset

Scan for bluetooth devices using following command to test the fix -

bt-adapter -d

Check your Wireless Card

After following the previous steps, the bluetooth should be working. If it doesn’t, the next step is to find any issues with your wireless module. In my case, my laptop has RTL8723BE which in the past has troubled me with many connectivity issues. Recent Linux Kernels are supposed to have a driver for this particular card.

This card supports two antennas but many manufacturers only put one antenna to save costs. When Linux tries to use both the antennas the failure occurs. The fix is to tell Linux to only use one of the antennas. To do this, run the following commands -

# Tell kernel to use antenna 1
echo "options rtl8723be ant_sel=1" >> /etc/modprobe.d/rtl8723be.conf
 
# remove the existing driver
sudo modprobe -r rtl8723be
 
# mount the driver again
sudo modprobe rtl8723be

If this doesn’t improve the wifi signal, try changing ant_sel to 2 and try again.

Another thing that can be done is to disable power management features of this card. This will help if you face frequent disconnections. To disable power management on this card, this StackOverflow post can be followed.

This should increase the bluetooth and wifi range of your card which may help in better connections. If the bluetooth still doesn’t work, continue reading.

Bluetooth driver failures

Another thing to observe is the dmesg output for any bluetooth related errors. In my case, I saw following errors -

sudo dmesg -w | grep -i blue
 
[    8.415745] Bluetooth: hci0: rtl: examining hci_ver=06 hci_rev=000b lmp_ver=06 lmp_subver=8723
[    8.415750] Bluetooth: hci0: rtl: loading rtl_bt/rtl8723b_config.bin
[    8.422228] bluetooth hci0: Direct firmware load for rtl_bt/rtl8723b_config.bin failed with error -2
[    8.422238] Bluetooth: hci0: Failed to load rtl_bt/rtl8723b_config.bin
[    8.422319] Bluetooth: hci0: rtl: loading rtl_bt/rtl8723b_fw.bin
[    8.461783] Bluetooth: hci0: rom_version status=0 version=1

Searching of this error on the web led me to one Arch Linux forum post which suggested to manually download the config binary file and put it in the right place. I have used artefacts from the lwfinger before, so I decided to trust this. But it is not usually recommended to download random files from the Internet and execute them. After downloading the file, I executed following command -

sudo cp rtl8723b_config /usr/lib/firmware/rtl_bt/rtl8723b_config.bin

Reboot the system afterwards.

Rfkill

However, this also didn’t work. Rfkill is a Linux utility used to manage wireless devices. Sometimes, a bluetooth device can be soft-blocked by rfkill. In that case, you will have to enable it before the bluetooth can be used.

yash@homelab ~> rfkill list
0: phy0: Wireless LAN
	Soft blocked: no
	Hard blocked: no
1: hci0: Bluetooth
	Soft blocked: yes
	Hard blocked: no

If any of your devices are soft-blocked, run the following command to unblock it -

rfkill unblock bluetooth

After this step, again check if the bluetooth can be enabled. In my case, even after this step, the bluetooth didn’t work, so I debugged further.

Udev

After a long debugging session, I discovered the following udev rule in /etc/udev/rules.d/ -

## DISABLE BLUETOOTH
SUBSYSTEM=="rfkill", ATTR{type}=="bluetooth", ATTR{state}="0"

I may have added this rule when I was trying to configure my system as a headless home server. Having this rule on a fresh system is highly unlikely, so you may not have to worry about this.

I removed this rule and rebooted the system.

Conclusion

Finally, after almost two hours of debugging, the bluetooth on my laptop was finally working. It is a story for another day that I didn’t use the bluetooth for even for 10 minutes after this.

Do leave a comment if you find this post helpful.

Train the algorithm

Sun, 12 May 2024 06:40:24 GMT

It is Sunday today so I decided to take a break from my screen-heavy routine. It rained yesterday night so the morning felt a lot cooler than the scorching heat I am used to in the month of May. The early morning breeze tempted me to go for a morning walk. Walks can be a great way to spend some time with yourself and contemplate on your life. I also could’ve this opportunity to have some time doing self-review. But I completely wasted this chance and spent most of my time staring at my smartphone and scrolling X (formerly Twitter).

I vividly remember another such incident that happened during the last year of college. It was March and I had just spent first few hours of my day scrolling through Facebook (it was still famous those days). An introspection on that incident prompted me to leave Facebook forever. I took several steps after that day to control my urge to use Facebook and it mostly worked. The decreasing popularity of Facebook among my generation definitely aligned perfectly with the task at hand and I was able to get rid of Facebook from my life.

Now, after almost six years, I have been battling yet another bout of social media addiction for last few months. Nowadays, I use primarily four social media apps - X, Instagram, Quora and WhatsApp. Below, I detail my usage patterns for these four apps and how I try to use these in a controlled manner.

Quora is the easiest one. Their content moderation policies and the subsequent boycott by prominent writers meant that the quality of the content available on the platform has greatly reduced. I benefitted the most from this. Now, I tend to open Quora once or twice a month at max.

WhatsApp is almost unavoidable for me. I have worked slowly over the years to reduce my usage of the app, to the disappointment of some people in my life. I rarely send messages to anyone and even rare is to get a message from someone. Now, if I receive a message, I am almost certain that the sender is in some emergency. WhatsApp’s recent changes (regarding business chats) has definitely increased daily notifications on my phone. Its unsubscribe functionality is mostly broken (looking at you, HDFC) so, I am not able to block some businesses. But I guess that is a nuance that I can live with, for now.

I started using Instagram more seriously during my courtship period, partly because my fiancée (now wife) was only active on Instagram. At that time, I was instantly glued to the Instagram’s reels functionality. It worked really well, showed me engaging reels. It somehow knew all my interests. I was spending 4-5 hours everyday just scrolling (or shall I say swiping?) through the reels. It even allowed me to save the reels for future, though saving reels mostly catered to my FOMO. Most of the saved reels remain untouched to this day. My usage of Instagram reached such a point that I started to experience several psychological and behavioural issues. I felt myself battling self-doubt, loathing and imposter syndrome. Seeing others’ artificial progress made me jealous. It gave me anxiety. The more I swiped, the less confident I felt about myself. The constant pressure to keep up with the latest trends made me scared and less aware about myself. The compulsion to announce to the world; birthdays, anniversaries and what not, the urge to upload that one photo out of the thousands, just to show the world that I am doing great in life. I am done with this manipulation, I am done with letting an algorithm control me. I am trying to eliminate Instagram from my daily life for a while now. Doing it suddenly would have probably resulted in Withdrawal Syndrome. So I decided to approach this issue in a stepwise manner. First step was to uninstall the Instagram app from my phone. I slowly forced myself to use the web version of Instagram. While the web version is functional, there are some hiccups here and there - stories take time to load, no doom-scrolling etc. These minor details definitely helped me to feel the irritation. Once the irritation kicked in, it was easy to stop using the web version also. Ultimately, I decided to take the final step of deactivation. It has been three days since I deactivated my account. I haven’t felt the urge to open the app even once. These are still early days and I need to keep a close eye on my behavioural patterns to avoid falling into the trap again.

My current go-to social media platform is X. X can be quite toxic if one is interested in certain topics - like politics. I tend to avoid these topics mostly. I use X primarily to gain knowledge about Tech, Indie-hacking, SaaS businesses and other similar topics. However, I am well aware about the potential pitfalls of using it more often so I consciously try to put limits on my usage of the web app as much as possible. Doom-scrolling is a definite no-no.

Having said this, I am not fully capable of cutting myself completely from the social media. We live in a world which is increasingly being controlled by the algorithm. The algorithm is constantly watching us, waiting to identify patterns in our behaviour. These algorithms are designed by some of the smartest people in the world who have spent years studying the human brain. It is difficult to beat them at their game.

In my opinion, it is okay to fall pray to the social media sometimes. But it is important to understand what you want and what you don’t. Tell the algorithm more of what you want. Spend some time understanding your interests and what values they bring. I find politics interesting, but it doesn’t add any real value to my skill set, so I try to not consume any political content on social media. Conversely, I work in tech, and I intend to get into solopreneurship, so I follow people doing great in these areas. Ruthlessly unfollow people who are posting frivolous content or content outside of your interest areas. Do this consistently and the algorithm will bend to your will.

If you are passionate about your goals and if you are willing to bite the bullet, nothing can distract you, not even the best algorithms in the world.

Homelab: The experiment begins

Tue, 27 Feb 2024 08:55:15 GMT

Background

I have been planning to set up a homelab for almost six years now. It all started in the last year of my graduation. At that time, the goal was to learn about PC hardware by building a high-end PC. On the other hand, I was already familiar with enterprise level hardware because of my involvement in the SSL lab upgradation effort (pity I didn’t get to feel that shiny new server I recommended as the next upgrade for the lab), so I also ventured into setting up a proper server but that was always a distant dream considering the cost to acquire a server and power it.

Later I started looking at Raspberry Pi. My earliest memory of the Raspberry Pi is from 2019 when I was working with Cisco in Bangalore. I did extensive research, prepared a plan but somehow ended up never buying the SBC. A Raspberry Pi is not sufficient alone in most cases - you need a case, an active cooler, cables, power supply, and a UPS. One buys all of this and suddenly the Raspberry Pi starts to look rather unattractive. I couldn't convince myself about the unit economics of buying the Raspberry Pi, so I dropped this plan.

Why now?

For the last 4-5 months, since I have decided to focus on my career again, I have been thinking about exploring the Linux world again. Naturally, the Raspberry Pi came back on my radar. I am going to work from home for the foreseeable future. More importantly, I have a mostly uninterruptible internet connection. The power connection sometimes fluctuates in my area. A RPi would instantly go down in case of a power cut, along with a high probability of data corruption. I was able to circumvent the issue of the router going down on a power cut by buying a router UPS, but buying one for RPi didn't make sense to me.

Being a backend developer, although the sysadmin, devops, networking, and platform skills don't bring any apparent advantages to my day-to-day work, I have decided to invest some time in home-labbing because -

It has been my passion for a long time and I want to finally experience it.
A peripheral knowledge of the infra and platform side of things definitely make it easier to design more efficient systems and debug non-trivial issues.
As I progress in my career, I will not be handling just one portion of the product. I intend to take more and more responsibilities. That means wearing more than one hat. A homelab provides me an opportunity to learn one such skill.

The Hardware

I bought a HP Pavilion Notebook - 15-ab516tx in the third year of my graduation. At that time, it was a pretty powerful laptop and served me well in all my Linux and Windows experiments throughout the college term.

It has been gathering dust in my closet since I started using MacOS and Apple hardware. Getting back to Windows and its keyboard layout has proven to be more difficult than I anticipated. So, my use of this laptop was limited to occasionally using Windows and as an offsite backup destination for some of my old data (that I am keeping only for nostalgia, huh). I was hesitant to keep it powered on 24/7 to preserve it for some more years. But then I convinced myself that in the worst case scenario, I will have to buy a new battery. Apart from that, the possibility of it getting fried is pretty miniscule.

It has an Intel Core-i5 6th generation CPU with four cores. It currently has 8 GB DDR3L-1600 SDRAM installed. The other RAM slot is a DDR type slot, but I am not sure if this is accurate. For now, I assume that I am limited to 8GB of RAM on this system. The ethernet port on this device is the Integrated 10/100 BASE-T Ethernet LAN.

Yet another under-utilised component on this device is the NVIDIA 940M (2 GB DDR3 dedicated). I intend to use it for video transcoding in future however, I am doubtful about the support for it. If this doesn’t work, I might have to settle for Intel’s inbuilt graphic card.

It came with a 1TB 5400 rpm SATA HDD. It got very slow with time, so I purchased a 512 GB SSD and placed it in the SATA slot of my HDD. Later I removed the disk tray from the laptop and installed the unused HDD into its SATA port using a caddy. So, now I have 1.5 TB of storage space. Windows 10 is installed on a 200 GB NTFS partition. Another 500 GB is occupied with my personal files (which I intend to organise later).

The OS

I have not tinkered with Linux for almost 6 years now. Since I left college, somehow my zeal to explore new stuff also diminished. I had developed a false impression that building something is more important than using what others built. While I still believe in it, I don’t want to restrict myself with such black and white statements. Both activities have their significance. What matters more is to keep the fire burning.

Anyway, I had a couple of choices for the OS on my server - Ubuntu server, Arch Linux, Debian minimal, and a few Virtualisation platforms - Proxmox etc. Proxmox was out of the question because I wasn’t sure that my laptop would be able to spin up more than a few VMs with just 4 cores and 8 GB of RAM.

I don’t have a good experience with Ubuntu in the past, even though it has good support for non-free drivers, especially NVIDIA. Arch Linux may have been a good choice but I wanted to stick with an apt-based distro. In the end, I decided to install Debian-minimal on a 100 GB dedicated root partition.

For some reason (most likely due to my RTL8723BE wireless card on my laptop), the installer failed a couple of times due to OOM issues. I had to boot up the installer by modifying the GRUB by adding following lines (although I doubt PCIE errors can cause OOM, but it worked, so all good) -

GRUB_CMDLINE_LINUX_DEFAULT="quiet pci=noaer"

Once the installation finished, I was greeted with the default TTY screen.

What’s next

The Debian minimal comes with literally nothing. Not even sudo. In the next post in this series, I plan to write about setting up the Debian server - networking, SSH, setting up lid-close action etc. Later I will explain how to set up Docker MacVLAN interfaces and make docker containers appear like another device on your network.

As with all other things, it is easy to get lost in many options available for homelab. So I have decided a few minimum required goals that I would like to achieve with this exercise -

Create a fully-reproducible setup.
1. Host a few of the interesting services - bookmark manager, Plex, Adguard, reverse proxy etc.
Allow outside world access to some of the services and secure the server.
Learn VPNs and configure one.
Document the progress and the concepts learned.
Few of the technologies I plan to learn -
1. Docker - I already have a good experience with it, so it would be a refresh for me.
2. Ansible - Automate the repetitive work
3. Wireguard - I think it would be a good experience to learn how it works behind the scenes. Also explore TailScale later.
4. DNS, Port Forwarding, CGNAT, IPv6, reverse proxy, cert management etc - to allow outside world access.

While this is an exciting thing to dive into, I will also be focusing on employing my server for development work. I have planned many exciting things for 2024, most of which will get some mention on this blog.

Stay tuned!

Thailand: Sandy Beaches to Bustling Streets

Sat, 19 Aug 2023 03:52:15 GMT

Introduction

Since COVID began, I have been at home, thoroughly immersed in work and routine life. I was never really an enthusiastic traveller, much less when nobody pushed me. I went on a few trips during college (mainly because of FOMO) and two visits to Coorg and Chikmagalur while in Bangalore. Laziness, finance and my tendency to stay clear of risks led to the state where I kept desiring to travel without concrete action.

Keeping this behaviour in sight, I always desired a partner who thinks differently from me when it comes to travelling. And God just listened to me. My spouse is a travel enthusiast (perhaps because she hasn't travelled much till now). So when she nudged me to go on a trip somewhere, I decided to break my bubble and take on the adventure.

I have never been to any place outside of India. So, a foreign trip was definitely on my mind for a long time. When planning, we considered many options - Bali, Sri Lanka, Bhutan, Malaysia, Thailand, and Singapore — all countries nearby to India.

At last, we settled on Thailand, partly because the flight tickets were the cheapest at the time of booking for Thailand. Another destination top on my list was Bali, Indonesia. However, the lack of a direct flight from Jaipur/Delhi to Bali meant that I had to spend some extra time on the flight (along with increased flight costs), so we decided to go with Thailand.

We had planned to visit Thailand for our honeymoon in January 2023, but we had to reschedule the trip due to unforeseen circumstances. We decided to reschedule it for June, which, although a low season in Thailand, has humid yet decent weather for roaming around. Meanwhile, we went to Manali for a short honeymoon trip.

Money Exchange

I did some research on popular payment methods in Thailand. For one thing, cash is still king there, at least for a tourist. Many shops accept QR code-based payments, and a few also accept cards. VISA and MasterCard rule in card space. The acceptability of Diners Club is pretty much limited to Airport lounges. My HDFC Diners Black Card was turned down everywhere I tried.

Anticipating this situation, I had already kept enough Thai Baht for day-to-day expenses. I got an exchange rate of 0.4 baht per rupee in Jaipur. In Phuket and Bangkok, I saw a few exchanges offering rates of 0.42 baht. Don't exchange your money at airports; instead, carry at least ฿4400 for Visa-on-arrival fees and a few hundred bahts to pay for transportation from the airport to your hotel.

Air Travel

I last boarded an aeroplane in March 2020 when I returned to Jaipur from Bangalore, seeing the signs of imminent lockdown. After that, I didn't get any chance to board a flight. So I was very excited about air travel again. Alas, my passport also got an opportunity to be useful to me after sitting idly for so long.

During this trip, I got to visit three airports - Jaipur Airport (JAI), Don Mueang International Airport (DMK) and Phuket International Airport (HKT).

The plan was to fully utilise Diners Black's unlimited international lounge accesses. However, due to hiccups by HDFC, the add-on card for my wife didn't arrive on time, so we had to settle with one Diners Black with unlimited lounge accesses and one DreamFolks card with only two lounge accesses. We decided to utilise only the two lounge accesses available to us. Both of these visits were at the DMK airport - one in the Coral Executive Lounge in the domestic terminal and the other in the Miracle Lounge in the international terminal. I was unsure if they would allow my wife as both cards had my name, but they decided to show two separate visits for me. Makes sense, as the cards were different.

Visa on Arrival

Thailand provides visas on arrival for Indian tourists. We reached DMK early morning at 5 AM. As we had a subsequent (but not connecting) flight to Phuket at 10 AM, we were confused about whether to take the Visa first or go to immigration first. Later we figured it out. If we had a connecting flight, we could have approached immigration and claimed our baggage first before getting a Visa at the final destination airport. But as our flight was not connecting, we had to get the Visa first. There was a long queue for Visa-on-arrival. It took us a good 1 hour to get our passports stamped. The fee is supposed to be ฿2000, but they took ฿2200 per person from us. The process is simple. You fill out a form and stand in a queue. When your turn comes, provide all your documents (including flight tickets, hotel bookings etc.), pay the fees and give your passport to them. After some time, they will come back outside with batches of stamped passports.

After getting our Visa, we proceeded to the immigration counter. Here also, the process was smooth. They took our fingerprints and photo and put an entry stamp on our passports. After this we claimed our baggage and waited for 2 hours to board our next flight.

Sim Card

Another thing that we researched was whether to use an Indian sim with international roaming activated or buy a local sim. International roaming packs of Indian telecom operators are quite expensive, so it was economical to buy the local sim.

I read on the Internet that the sim providers at airports sell sim for a premium, so it is better to buy from the city. My experience is different. We didn't buy the AIS sim at the airport and waited to reach Phuket city. Inside the city, we didn't find any vendor selling AIS sim. Everybody was selling dTac sims. The pricing was the same everywhere, ฿299, but the benefits provided were different. At last, we decided to settle on a travel sim by dTac, which needed no documents and provided 50 minutes of calling and (not truly) unlimited internet for 7 days. This was more than sufficient for our daily needs.

Phuket

Several transport options are available at the Phuket airport to reach your hotel. The private taxi costs ฿800 for the Patong area, and a shared minibus costs ฿180 per person. We opted for the minibus because it was economical and didn't compromise on comfort either. Around ten people can sit in a minibus, and the driver drops you at the hotel.

Day 1

We reached our hotel, The Silver Resortel, at around 2 PM. I booked this hotel mainly because of the excellent reviews on booking.com and its proximity to Patong Beach and Bangla Road. The hotel was equipped with all the necessary amenities and facilities. Like many other hotels in Phuket, they asked for a security deposit of ฿500 which was refunded at checkout.

It has been 5 years since I stepped into a beach town. So, the first thing I did after a quick nap was to run towards the beach. We forgot to take the free beach towels from the hotel, so I had to sit in the sand. In a way, it was a good thing. The sea was rough in Phuket as it is the rainy season there. It was first beach visit for my wife, so she was equally excited. We sat on the beach for 2 hours, played in the pristine, chilled water and thoroughly enjoyed the sunset.

After coming from the beach, we decided to search the area for a good Indian Vegetarian Restaurant. While roaming around, we reached the famous Bangla Road, the party point of Phuket. And boy, what a spectacular scene this was! People were wandering around freely. Bargirls and ladyboys wearing skinny clothes try to lure customers into the clubs. Open clubs playing music, customers having "deep" conversations while sipping beer, a lot of vendors selling every kind of stuff and people performing on the road. Being in such an environment was a new experience, but I did not feel intimidated or threatened by anyone.

In our search for food, we reached Amritsr Restaurant and ordered Kulchas. We weren't expecting good taste, but we were surprised. The food was tasty, though somewhat bland. The waiters and manager treated us nicely, and we left with a good impression. Throughout my trip, I did one thing wrong - converting Thai Bahts to INR every time I decided to make a purchase. Comparing the prices in Thailand to their exchange rate pricing in India was incorrect. This made me conscious as I found everything expensive compared to India. We spent ฿180 at Amritsr, which seemed very expensive for a Kulcha. But we were to get more surprises in days to come.

While planning for the next day, we were confused about tours to Phi Phi Island or James Bond Island. After watching some Youtube videos, we decided to visit Phi Phi Islands this time and leave James Bond Islands for another trip. While travelling back to the hotel from Amritsr restaurant, we stopped at a few tour agents and queried about the tour plans. The thing with these tour operators is that they quote exorbitant prices. It is always advisable to bargain and try to bring the costs down.

The first agent gave us a quote of ฿3600 for a two-person trip. After haggling with her, she pretended to talk to her boss and agreed to ฿3000. We were unsatisfied with this price, so we decided to try elsewhere. The last agent we tried finally gave us a price of ฿2500 for two people. On talking to him about Phuket City Tour, he agreed to a total cost of ฿4600 for two persons. So that was what we could bargain. ฿1250 per person for Phi Phi Island Tour and ฿1100 per person for the City tour. We tried to negotiate a bit more, but the agent did not relent, so we decided not to pursue further.

Day 2

The next day started early. The tour started at 7.30 in the morning. The minibus picked us up from the hotel, and we were transferred to the eastern coast of Phuket, where we were provided breakfast. As vegetarians, we had to satisfy ourselves with fruits.

After breakfast, we boarded the speed boat along with 30-35 people. This was my first sea ride, so I was quite excited about it. We visited 4-5 islands throughout the day and stayed on the boat for over 5 hours. We had lunch on Phi Phi Island, a fully commercialised island with many resorts and restaurants.

Our tour also included Snorkelling, but we weren't confident about it, so we decided not to do it. I am definitely going to try it someday in future, though.

After a gruelling day at sea, we came back to our base at around 6 PM. From there, it took another 1 hour to reach our hotel in the Patong area.

For Dinner, we decided to visit Hare Krishna Vegetarian Indian Restaurant. This was one of the most economical yet satisfying restaurants we tried on this trip. The food was good, the hospitality was great, and the bill was pocket-friendly.

On our way back to the hotel at night, we decided to go back to the tour agent from whom we took the Phi Phi Island tour and booked the Phuket City tour package for ฿2100 for two pax.

Day 3

The city tour was scheduled from 10 AM to 4 PM with no lunch included. We visited Tiger Park, Big Buddha, Chalong Temple, a Cashew Nut factory, a Honey farm and Elephant trekking.

The Tiger Park has many tigers ranging from big, medium and small sizes, which are domesticated in such a way that you can go near them, touch them and pat them. I don't know if they were drugged, but I didn't feel good seeing them. I didn't see any visible signs of brutality there, so I don't know if it is just the result of their upbringing in human vicinity or the effect of any drug.

Big Buddha has a very large idol of Buddha and a small temple dedicated to Buddha. It has some good scenes. Apart from that, I don't see any significant value in this place.

The next stop was the Chalong Temple. This is another one of several Buddhist temples we visited on this trip. There are many temples in this complex, all of which were alien to us. So we did what a Dharmic believer would: bow and pray to the idol.

After this, we went to a Cashew Nut factory and Honey Farm. It was likely the collaboration of our tour operator with these two enterprises to help increase their sales to foreign tourists. The products were of good quality and unique because you probably won't find similar things anywhere else.

After these, we went to the last stop - Elephant Trekking at Nopparat Elephant Camp. I regret going there. I could clearly see overworked elephants and their mahouts using a sharp objects to pierce them and keep them on track. However, at that time, I enjoyed the experience of sitting on an elephant. The mahout even allowed me to freely sit in his place and ride the elephant. This was risky and could have resulted in an accident, but nothing wrong happened. I would not support them by going there again. But the other side of this story is that this money is the only way the elephants could survive while being in captivity.

This marked the end of our city tour. We were dropped off at our hotel by the tour operator. After taking some rest, we decided to head to the beach to enjoy the vastness of the ocean for one last time for a while.

After returning from the beach, we decided to go to another vegetarian restaurant - Kailash Parbat. We found it to be okayish despite its high ratings on Google.

After dinner, we booked our minibus to Phuket airport, which cost us ฿200 per pax. We had to board our minibus early the following day; we decided to bid farewell to Phuket and left for the hotel after roaming around Bangla Road for an hour.

I found Phuket to be a friendly and safe city to roam around. We missed visiting many things - the most prominent being Phuket Town. However, we thoroughly enjoyed whatever we could see. The local people we interacted with were humble and behaved patiently and calmly with us. Being a tourist city, people will try to coax you, so one needs to be alert constantly; overall, the people appeared friendly.

Bangkok

Before I start describing the trip, let me spend a moment praising Bangkok. I have not travelled extensively, so my experience is limited. I haven't seen any city more modern than Bangkok. I feel pity for the condition of Bangalore when I compare it to Bangkok. Even if I compare the poshest portions of both cities, there is simply no comparison. Everything was absolutely top-notch - the roads, the traffic management, the comfort of pedestrians, and the traffic infrastructure. I especially loved the public transport of this city. It was functional and well-connected with each other. All three modes - buses, MTR and BTS function together in tandem to serve the city, which also helps in relieving the traffic on roads. Absolute beauty!!!

Day 4

The next day we reached Phuket airport to board our flight to Bangkok by 7.30 AM. We had to wait at the airport till 10 AM for the flight to take off. We reached Bangkok by 1.30 PM and took a Grab cab to our hotel. I liked the interface of Grab more than any other ride-hailing app I have used till now. It was easy to use and smooth (looking at you, Uber). The ride cost us ฿310, which is not that high if I compare it to the bus fare of ฿116 I paid while returning to the airport later in the trip.

The hotel I booked was the Lyf Sukhumvit 8 Bangkok. The hotel looks nice on paper, but the lack of basic facilities and the farce of shared spaces (I assume this is to save costs) made it a not-so-value-for-money hotel for us. The room was tiny, particularly for a couple. There was no telephone, fridge or wardrobe in the room. You need to go to the reception to get all the necessary daily use items like towels, comb, toothbrushes, pen etc. It is usual for 3-star hotels to not provide these, but I haven't seen a 4-star hotel not providing them anywhere in the room. The bathroom was shared one - meaning that you can't use the toilet and bath area simultaneously because the door is sliding without locks.

However, there were some positives as well. The interior of the hotel was very innovative. The ideas of a shared kitchen and shared workspace on the ground floor were unique.

As we left Phuket quite early, we didn't get a chance to have breakfast. So we were starving when we reached Bangkok. We decided to explore the hotel later and find a restaurant for quick lunch. We found Dosa King to be very near, so we proceeded to have lunch there. We liked this restaurant so much that we came here twice after this visit. The food here was excellent, the ambience was good, it didn't hurt our pockets much, and the owners were amiable.

As we were filling our stomachs, we started looking for a nearby mall for a quick visit. And we found one at walking distance - Terminal 21. It is a massive mall with many shops selling various items, primarily apparel and related accessories.

Day 5

The next day, we decided to roam the city on our own. We had a list of monuments ready to visit - Wat Pho, Wat Arun, Grand Palace, Emerald Buddha, and Wat Saket. The idea was to explore public transportation in Bangkok.

We first boarded the MTR to reach Wat Pho. It was my first experience travelling on a metro train. I never found any need to travel in a metro in India. Bangkok has two types of metro - the overhead BTS and the underground MTR. We reached an MTR station and got the token by paying at an automatic vending machine. Next, we boarded the train, which was crowded, but still, the people organised themselves nicely.

The day was very humid, and the sun was shining brightly. We were drenched during our short walk from the MTR station to Wat Pho. But alas, we reached there. The entry fee to Wat Pho was a whooping ฿200 per person, which is too high. But then I realised the entry fee for foreign tourists to enter historical monuments in Jaipur. And it all started making perfect sense.

There are many Buddhist temples on the premises of Wat Pho. We got bored walking around because we didn't opt for a guide. Nothing made any sense as we were not familiar with the history. But I guess, even with a guide, my feelings would not have been any different.

After our experience of Wat Pho, we decided to rethink our visit to Grand Palace. Seeing the considerable entry fee of ฿500 and an expectation of an experience similar to Wat Pho, we decided it was best to go to something other than Grand Palace. But mind it, the Grand Palace is the most visited tourist attraction in Bangkok, and if you are interested in museums, you should definitely see it.

Next, we decided to visit Wat Arun across the Chao Phraya River. It was a long walk to reach the boat ferry. It is yet another overcrowded Buddhist temple with people doing photoshoots in traditional Thai costumes.

From Wat Arun, we decided to go to Wat Saket. To reach Wat Saket, we had to take another ferry to go back to the other side of the city and then board a bus to Democracy Monument and then walk from there.

While we were walking towards Wat Saket, we found another attraction - the Statue of King Rama III. It was a nice little resting point for us with some good photo points.

We wanted to go to Wat Saket as it is termed a good sunset point by many online blogs. Unfortunately, when we reached there, the weather had turned cloudy, and the sun was nowhere in sight. We waited there for 2 hours for the sun to show up, but this didn't happen, and we had to return disappointed. Nevertheless, we clicked some lovely pictures.

After Wat Saket, we decided to call it a day, and returned to our hotel after having dinner at Saras Veg Food. This was our most expensive dinner on the trip, but I found the food average.

While walking to Saras Restaurant, we stumbled upon an Ardhanarishvara statue on a busy street. I was pleasantly surprised to see this. You perhaps won't find this iconography in many places in India.

Day 6

The next day of our trip was solely reserved for mall visits. Bangkok is known for its huge malls, and we decided to spend one day visiting them. We didn't have anything specific in mind for shopping. We visited the Platinum Mall, the Palladium Mall, The City Complex and The Market. We completely missed Central World, the largest mall in Bangkok. It never came up in our research somehow. We will visit it someday, hopefully.

After having dinner at Maa Pure Vegetarian Indian Restaurant, we decided to walk to the bus stand. On our way, we passed through Baiyoke Night Market, which is a wholesale cloth market. I saw a lot of Indians shopping there. As there was a light shower, we decided to leave the place and reach the bus stop quickly.

Day 7

The day brought an end to our visit to Thailand. The next day, though our flight was for 7.30 PM in the evening, we packed our luggage, had an early lunch at Dosa King (again) and left for DMK airport by around 1.00 PM to avoid paying the hotel for one extra day. We had to wait outside airport security for 3 hours before our luggage was checked in. Once we cleared the airport security, we decided to go to Miracle Lounge and sit until our boarding started. However, our flight got delayed due to technical issues, and we reached Jaipur Airport late by 2 hours.

Cost Breakdown

We jumped our initial estimation by around 10%, primarily because of the costs incurred due to the rescheduling of flights. As there were four flights in our itinerary, the costs increased significantly. We could have planned it better by booking Jaipur to Phuket and Bangkok to Jaipur flights. But Jaipur to Bangkok return flights were cheaper when we booked them initially.

Our next significant expense was on hotels, food and activities. I found The Silver Resortel for a reasonable price, but the decision to opt for Lyf could have been better. We spent most of our daytime outside the hotel, so we could have stayed in a 3-star hotel for a much lower price.

We wanted to eat in vegetarian restaurants only. And vegetarian restaurants in Thailand are rare. Due to this, the cost of eating in such restaurants is high. As the demand is high, restaurants serve non-vegetarian food to every stratum of society. But, these Indian vegetarian restaurants have to aim at tourists only due to less local demand. So accordingly, their prices are high as well. We had to spend around ฿1000 a day on food, despite eating homemade thepla for breakfast for the first 3 days of our trip.

Activities in Phuket were another significant expense. These could not be avoided, and by going in the low season, we saved quite a bit on it. According to some YouTube blogs, what we did for ฿1250 requires at least ฿1700-1800 in the regular season. So definitely, we saved a bit here.

We found the shopping to be expensive there. I don't know why people go to Bangkok to buy stuff. I found everything to be costly by Indian standards. It may be futile to convert everything into INR, or perhaps people want to feel nostalgic about Bangkok by buying stuff. Nevertheless, we didn't purchase many unnecessary things and spent only on souvenirs for friends and family.

Conclusion

We missed visiting many attractions in Bangkok and Phuket, most notably Phuket Old Town, James Bond Islands, Karon Beach, Kata Beach in Phuket and Central World and various Night Markets in Bangkok. Still, there will always be such things in a travel plan. You simply can't cover everything.

Nevertheless, Thailand was my first international trip, which will always be memorable for this reason alone. I loved visiting this amazing country, and the hospitality of the Thai people really won me. I plan to come to Thailand again to explore its northern ends soon.

India turns 77!

Tue, 15 Aug 2023 16:25:24 GMT

India is celebrating its 76th independence day today. Irrespective of all the political gimmicks (like per capita GDP vs total size of economy), there is no doubt about India’s economic progress and cultural awakening in the present times. However, we must be careful not to turn in religious bigots. This is not the characteristic of our culture, at least for the last millennium.

With all the negativity surrounding us in the current environment, I wonder if a symbolism like Independence Day is of value anymore. There used to be a time when my school used to have a function on two days of national importance. There used to be intense preparations for the day for the preceding one month. On the day of the event, we used to get ready early and leave for school in anticipation of some cultural activities, prize distributions, dreaded Yoga activities, and not the least, laddus! There used to be some energetic patriotic sermons from local leaders and persons of eminence. It was an excellent introduction to cultural patriotism for us kids, away from all the intricacies and complexities of the culture, nation, and the ultra-modern concept of patriotism. I can’t help but reminisce about the time gone by.

Yesterday, I was talking to my cousin, who has two kids. He mentioned that there is no cultural activity in his son’s school this time. School administration no longer wants to indulge in such activities due to mounting expenses. I couldn’t help but wonder how the child will develop a sense of patriotism (it is another debate if creating such a feeling is necessary or not). If the state is no longer interested in teaching the feeling of belongingness to the nation, how will future citizens behave when they have the command? There are many reasons for people to break away from this nation; why do we want to create one more? Politicians are busy with their agenda. When I opened a Hindi daily today, I found only the advertisements for Government’s latest schemes. They find it enough to mention Independence Day in passing.

People still curse the English for what all wrong was done to Indians for 190+ years till 1947. I don’t understand the nostalgia. What happened was wrong, and we must ensure that history doesn’t repeat itself. But complaining about it will not benefit us as a nation. Nowadays, it is commonplace to bash people of the past for all that is wrong with India today. I am a firm believer in the fact that one should not judge the past through the moral yardsticks of the present. Using history to justify today’s politics won’t benefit India. Indians as a people have evolved a lot since 1947. What was right in 1947 may not be right in the present. The decisions leaders took at that time may not be consistent with the facts available today. They must have made a lot of wrong decisions. They made some blunders. They were wrong in a lot of their judgments. But does it matter anymore? Can we pull an Endgame-like Time Heist and put things back in place? Why do we still cry over what happened 75 years ago if not? People talk about Akhanda Bharata and all the nostalgia surrounding it. Suppose all the people today living in the so-called Akhanda Bharata areas come back to join us. Is India capable enough to sustain herself and not die beneath the weight of religious bigotry? The advent of the Internet and social media seems to have aggravated the religious zeal that is unlikely to calm down in times to come.

Despite all the flaws in India as a nation and political system, I am very optimistic about its future. The country has done great on many of the UN’s sustainable development goals. There is much to achieve, but we are moving in the correct direction. Reservation system, religious attitudes of people, and shady politicians fuelling all kinds of partisanship in the people are the biggest threats to India and her future. While I understand the atrocities done on certain sections of Indian society in the past, I wonder if the time is ripe for overhauling the reservation system to make it more beneficial for needy sections of the society. We seem to be moving in another direction where the more the population of a caste, the more reservation it will get, irrespective of the political clout and financial status of its people. I believe this is a recipe for disaster in times to come. It is the age of science and technology. Suppose people get to the upper echelons of the scientific and technical community based on their castes and not their academic performance. In that case, we may be delayed in getting our rightful place (of Vishwa Guru, as the RW likes to term it) in the world. I know that with this biased opportunity system, we have brought many previously deprived yet talented people to the forefront, and education definitely improved their conditions generation after generation. Similarly, perhaps a few generations later, we will reach the top of the world order. But will we have the kind of demographic dividend then? I am doubtful.

Until next time 🇮🇳🇮🇳🇮🇳

Add reading time to blog articles

Sat, 05 Aug 2023 17:57:24 GMT

This article will show how to add estimated reading time to your posts. This is pretty easy to achieve with Contentlayer’s computed fields.

This article assumes that you have read my previous post and integrated Contentlayer into your project accordingly. We will build upon the foundations laid in the last article and add a new computed field to the Post document type to display the estimated reading time of the article to the readers.

How to compute reading time?

Computing the reading time of any text content requires two things -

The text content itself, which, thanks to Contentlayer, is available to us in raw form in post.body.raw where post is the JSON parsed by contentlayer from the mdx file.
An algorithm to calculate the reading time

We can come up with our own algorithm or use a package to do the heavy lifting for us. A number of packages exist on the NPM registry to help with this task -

read-time-estimate appears to be a more complete package that also takes image read times into consideration, but I am not sure if it is meant as a ready-to-use package. At least I couldn’t get it to work with my setup. So, I decided to go with reading-time, a far more popular package.

Adding a new computed field

After finalising the package to use, it’s time to install it. I will use yarn to install the package. You can find a similar command for your package manager.

yarn add -D reading-time

This will install reading-time as a dev-dependency in our project, which is fine given that we will only be utilising this package at build time.

Once installed, it is time to import and use it in the contentlayer.config.js -

import { defineDocumentType, makeSource } from "contentlayer/source-files"
import readingTime from "reading-time"
 
const Post = defineDocumentType(() => ({
  name: "Post",
  filePathPattern: "posts/*.mdx",
  contentType: "mdx",
  fields: {
    title: { type: "string", required: true },
    published: { type: "string", required: true },
    description: { type: "string" },
    status: {
      type: "enum",
      options: ["draft", "published"],
      required: true,
    },
  },
  computedFields: {
    slug: {
      type: "string",
      resolve: (doc) => doc._raw.sourceFileName.replace(/\.mdx$/, ""),
    },
    readingTime: {
      type: "json",
      resolve: (doc) => readingTime(doc.body.raw),
    },
  },
}))
 
export default makeSource({
  contentDirPath: "content",
  documentTypes: [Post],
})

Rendering it

In the generated JSON files, we will have the readingTime property available for use in the components.

{
  "title": "Hello World",
  "published": "2023-07-30T13:05:24.000Z",
  "status": "published",
  "body": {
    "raw": "\nHello world! This is my first post.\n",
    "code": "var Component=(()=>{var sr=Object.create;..."
  },
  "_id": "posts/hello-world.mdx",
  "_raw": {
    "sourceFilePath": "posts/hello-world.mdx",
    "sourceFileName": "hello-world.mdx",
    "sourceFileDir": "posts",
    "contentType": "mdx",
    "flattenedPath": "posts/hello-world"
  },
  "readingTime": {
    "text": "1 min read",
    "minutes": 0.035,
    "time": 2100,
    "words": 7
  },
  "type": "Post",
  "slug": "hello-world"
}

To use it in the component, we can use the post.readingTime.minutes property to show the time-to-read, something like this (here post contains the JSON data shown above) -

<p>Time to read: {Math.round(post.readingTime.minutes)} mins</p>

Half a Decade in Tech

Thu, 03 Aug 2023 10:18:46 GMT

I completed five years in the tech industry on August 1. I started out as a fresher straight out of college. I had no idea how the journey would turn out to be. I have sailed smoothly through the last five years by God's grace. Although I sometimes feel left behind compared to my peers in career growth, my personal metrics of personal development have largely been on the positive side (worth mentioning that I have evolved the way I look at personal growth as well.)

Often, I struggled with the feeling that I was not fit for this industry. My peers are doing great in their roles, achieving promotions year after year, running side gigs, moving to the US - earning in dollars and getting exposure to the Bay Area. I felt like missing out on great things in life. Whenever I open Instagram or LinkedIn, I see people enjoying their lives to the fullest. Of course, I know that what I see may not be the reality. Still, I couldn't help but doubt my capabilities.

That's how life plays out. Some days are bright and sunny. On these days, you feel like that king who can't do anything wrong. On the other hand, there are those gloomy days when you wake up doubting yourself. Nothing goes your way. You feel pathetic, and helplessness clouds your judgement. It is on these days when you will find the real you. How you handle these days will make or break you.

I have become a different person from what I was five years ago. The tech industry has transformed me to a great extent. I am more confident, concentrated and mature. I am rediscovering the long-missing criticality in my thought process. I have learnt to be satisfied running the race at my own pace. I am working slowly but steadily on my future. And I am proud to claim that I am better mentally and physically than five years ago. I believe this is the most appropriate criterion to determine my success.

Coding is the easiest

When I was in college, coding felt like the most challenging task. I was afraid that I would not be able to manage the expectations of my future teams. Turns out, I was wrong. Writing code is easy, maintaining it is harder and convincing people the hardest.

Maintaining the code takes a lot of effort and patience if not done correctly right from the first commit. Customer demands and internal requirements change over time. Your team changed its size and shape, and the persons who wrote that library are no longer with the company. Without proper documentation and UTs, it gets difficult for newcomers to take ownership without an upfront investment of significant human hours. Unfortunately, this is how it works in most tech teams, and the only way to get the work done is to get your hands dirty with code.

One of the best compliments I received was from one of my managers. He told me -

On my commits, QA doesn’t have to worry about doing deep validations because of the extensive testing I perform on my code.

In fact, in my last 1.5 years at Netskope, I have been responsible for only two regressions - the first happened in my first week at the company and another a year later because I was unaware of that particular code path.

People are the toughest nuts to crack. Early in my career, I used to try to convince my seniors at face value without any research. Over time, I learnt that it is easier to convince people when you have data. Substantiate your claims with solid data. They will listen. The seniors, who, I thought, were hurdles in my growth, were just trying to do their jobs. I have grown strong empathy for the management in these years. As a result, I find it easy to talk with them after I have done my due diligence and collected enough data to support my propositions. Unlike code, people behave differently according to the situation. The faster one adopts to handle people, the faster one grows.

If you don’t ask, you don’t get

In a typical tech team at a reputable company, everybody is a top talent. Everybody wants to reach the next level; hence the competition is steep. When everybody is trying their best to achieve that next promotion, it is difficult to get it without non-work effort. That is where the people skills come in handy.

My feeling of FOMO comes from the fact that I never sought any kind of promotion or pay raise at work. I always trusted my managers to do the heavy lifting, which never worked. If I don’t ask for it, it doesn’t mean nobody else does too. And the managers prefer those performing at my level and have also explicitly requested promotion. I have been trying to change this for the last few months. Being vocal about your work is a necessity to reach the next level.

Customer first

Businesses need to generate a profit. No matter how world-class my product is, whether you use cutting-edge tech in developing your products, customers don’t care. They don’t care if you are following all the best practices in the world. They only care about getting their work done; if your product is not helping, it is useless for them.

For a business, the latest and greatest doesn’t matter as much as getting the shit done. If you can’t deliver on time, you will lose the race. Customers love quality but also want to see things working as quickly as possible. It is not that your work is over once the product is shipped. After deployment, subsequent support is equally important. There are SLAs to honour, bugs to be fixed, and those weird customer requirements you couldn’t guess during those first architecture design meetings. Things often go wrong, and you must fix these to win that new contract next year.

Is Work-Life Balance a myth?

I have been fortunate to have jobs with an excellent work-life balance in the last five years. Rarely I had to work on weekends. Work does get spilled to after-work-hours at least once every few days. However, even then, I get a lot of free time. But I did not utilise this time properly. If I had used my time correctly, I could’ve learnt a lot of new stuff (hobbies and work-related).

My focus in the last five years has been on various non-tech things. For some reason, staying in touch with the latest tech never became my top priority. It is only in the last few months that I have started to explore again, perhaps because I have realised the importance of it for the next-level transition. As a side-effect, I expect an improvement in the activity on this blog as well.

On-calls are another aspect of a job at some software companies. I got introduced to it at Netskope, where every developer has to be on-call for one week continuously once every two-three months. Being on-call comes with a specific stress level. Still, it is a great learning opportunity to get to know the product from a critical standpoint. Although I don’t enjoy it very much, it is part and parcel of the job, so I can’t ignore it either.

Remote work is the future but …

Since the pandemic, remote work has picked up pace. I have been working remotely for the last three years, and while I enjoy it, I am concerned about my career growth. Looking at the state of big tech, I am uncertain for how long the rest of the industry will be willing to continue with remote work. On the other hand, due to the resistance shown by the big tech towards remote work, there may be a significant drift of talent towards more-remote friendly companies.

Remote work suits well to someone like me, who is appalled by the pathetic state of infrastructure and quality of life in the major Indian tech cities. Unless the state of affairs in these cities improves, I find it difficult to even think of moving back. However, having a job is the prime concern, and any probable threat to my job because of remote work may be a compelling reason to consider moving back. But until then, Jaipur is the best.

There is still a long journey left for me in the industry. Five years in tech is like a lifetime. It sometimes becomes challenging to keep up with the rapid changes in the tech world. With the advent of AI, it may become even more difficult for knowledge workers to stay relevant. Instead of running behind every latest framework, sticking to the basics would be a good idea for me. Frameworks may come and go, but basics will always stay in fashion.

Hoping to be at my best in the years to come 🥳

Image Component in Next.js

Sun, 30 Jul 2023 13:10:24 GMT

RSS feeds in Next.js blog

Sun, 30 Jul 2023 13:10:24 GMT

Integrating Contentlayer with Next.js Blog

Sun, 30 Jul 2023 13:00:24 GMT

The second part of this series explores Contentlayer and how to integrate it with a Next.js blog. I will elaborate on my experience managing content on my blog throughout the last eight years using various tools, why I decided to use Contentlayer and how to integrate it into a Next.js blog.

History

When I was starting out on my journey with Next.js to build the next version of my blog, I had to figure out a way to feed data into my application. I used only traditional static site generators like Jekyll and Hugo until then. SSGs work in this way - One writes the content in local markdown files; the SSG takes these files, extract the front matter and actual body, converts the extracted information to variables and feeds them into the actual server-side rendering logic, which will generate the static HTML files. Storing these content markdown files in Git is what is today called the Git CMS. I am a big proponent of keeping the content in a Git repository because it makes it easier to track the writing journey without tying too much into a vendor.

Although the concept of SSGs came into being with the release of Jekyll in 2008 (at the same time as Github and its iconic Pages feature), I actually started my journey with it in 2015. Since then, technology has advanced by leaps and frogs, and several new approaches to managing content and code have emerged.

I also dabbled with integrating headless CMS (WordPress, Sanity and Contentful) and GraphQL based Gatsby at various points in past few years, however I was never successful because of my unease with JavaScript. My unwillingness to rely on any third-party vendor for storing my content and the costs associated with self-hosting any open source CMS also contributed greatly to my failure to transition.

Migrating to Next.js

When I was starting to migrate my blog to Next.js, I had a couple of requirements in mind -

I preferred using a headless CMS for storing the content. A few reasons are - a. My writing was already suffering because of my over-emphasis on coding instead of writing. b. I was somewhat frustrated with my blog's overly complicated publishing process.
The CMS should allow export in popular open formats when I decide to migrate in future.

Notion fits nicely into my requirements. It has a well-defined (although with poor performance) API. It is a popular tool with the developer community and offers the export of the content in markdown format. Notion would have worked well for my requirements, but I couldn't integrate it properly with my blog due to my lack of JavaScript experience.

So, I reverted back to using markdown for my blog's content. I still write in Notion's excellent editor but then export the file to markdown and manually edit image paths etc., to work with my blog's structure.

However, using md and its more advanced sibling mdx with Next.js is tedious and error-prone. A bunch of tools need to be integrated to make MDX work with Next.js. unified, remark and rehype ecosystem provides some great tools, but combining these is not trivial. I had already wasted enough time struggling with Notion. I might have abandoned the project altogether if I had wasted a few more days.

My search finally ended on Contentlayer. Contentlayer is a tool which transforms unstructured markdown content into structured type-safe JSON data structures and provides accessible interfaces to access these data structures. This means I could import all my content in the code files using an import statement and work with the data as if working with a Typescript interface. This eliminated the need to struggle with additional boilerplate code needed to use the low-level libraries to interact with the MDX files.

With that in mind, let's see how to integrate Contentlayer in our Next.js blog. In later posts of this series, we will also see how to utilise the extensive plugin ecosystem provided by the unified project to enrich the data interfaces exposed by Contentlayer.

Install and Configure Contentlayer

Contentlayer's getting started guide is a good resource to setup your Next.js project to work with Contentlayer. I have described the same steps below for quick reference -

First, add contentlayer and next-contentlayer to the project -

yarn add contentlayer next-contentlayer

The next-contentlayer provides an API to stitch together next and contentlayer by hooking into the next dev and next build commands. This will ensure that contentlayer runs automatically for local development and production deployments.

Now, we need to patch the next.config.js to use the API provided by the next-contentlayer to hook into the commands mentioned above -

// next.config.js
const { withContentlayer } = require("next-contentlayer")
 
module.exports = withContentlayer(nextConfig)

Content Schema

Once we have contentlayer setup, it is time to define the structure of our data that will be used by Contentlayer to parse the MDX files. Contentlayer can also process the parsed content to derive new fields. Let's see how it is done.

Create a new file, contentlayer.config.js, in the root of your project. Inside this file, I am defining a Post document type -

import { defineDocumentType, makeSource } from "contentlayer/source-files"
 
export const Post = defineDocumentType(() => ({
  // The name of the document type
  name: "Post",
 
  // The type of the files to parse. 'md' also works
  contentType: "mdx",
 
  // The path of the mdx files, relative to contentDirPath
  filePathPattern: "posts/*.mdx",
 
  // Fields present in the frontmatter of the MDX file
  fields: {
    title: { type: "string", required: true },
    published: { type: "string", required: true },
    description: { type: "string" },
    status: {
      type: "enum",
      options: ["draft", "published"],
      required: true,
    },
  },
}))
 
export default makeSource({
  contentDirPath: "content",
  documentTypes: [Post],
})

In the code snippet above, the fields key is used to store the custom fields related to the metadata of a post, such as title, publish date etc. Any field can be made mandatory by using required: true.

Now, create a folder called content in the project's root, and inside this folder, create a subfolder called posts. We will store the mdx files inside this folder. Let's start our first post, content/posts/hello-world.mdx.

---
title: Hello World
published: 2023-07-30T13:05:24.000Z
status: published
---
 
Hello world! This is my first post.

If next dev is running in the background while you save this file, you will see the following output in your terminal -

wait - compiling...
File updated: content/posts/hello-world.mdx
Generated 1 document in .contentlayer

Contentlayer Artifacts

When next dev is run for the first time after setting up Contentlayer, Contentlayer creates a hidden folder inside the project root directory - .contentlayer. All the processed files, generated data structures, and helper functions and interfaces are stored here. Let's see the contents of this directory -

❯ tree
./
├── .contentlayer/
│   ├── generated/
│   │   ├── Post/
│   │   │   ├── _index.json
│   │   │   ├── _index.mjs
│   │   │   └── posts__hello-world.mdx.json
│   │   ├── index.d.ts
│   │   ├── index.mjs
│   │   └── types.d.ts
│   └── package.json
├── content/
│   └── posts/
│       └── hello-world.mdx
├── contentlayer.config.js
├── next.config.js
└── tsconfig.json

Contentlayer converts the static MDX file (line 15) into a structured JSON file (line 8). All such JSON files are accessible through the allPosts variable exported from index.mjs on line 7. In addition, it also generates type definitions (line 11) which helps Typescript understand the interface provided by the allPosts import.

// .contentlayer/generated/Post/posts__hello-world.mdx.json
 
{
  "title": "Hello World",
  "published": "2023-07-30T13:05:24.000Z",
  "status": "published",
  "body": {
    "raw": "\nHello world! This is my first post.\n",
    "code": "var Component=(()=>{var sr=Object.create;..."
  },
  "_id": "posts/hello-world.mdx",
  "_raw": {
    "sourceFilePath": "posts/hello-world.mdx",
    "sourceFileName": "hello-world.mdx",
    "sourceFileDir": "posts",
    "contentType": "mdx",
    "flattenedPath": "posts/hello-world"
  },
  "type": "Post"
}

Below is the generated type definition for the Post document type -

// .contentlayer/generated/types.d.ts
 
/** Document types */
export type Post = {
  /** File path relative to `contentDirPath` */
  _id: string
  _raw: Local.RawDocumentData
  type: 'Post'
  title: string
  published: string
  description?: string | undefined
  status: 'draft' | 'published'
  /** MDX file body */
  body: MDX
}

Computed Fields

Once Contentlayer has generated the JSON files, the content is ready to consume on our pages. We will use Next.js's dynamic routes to render a single blog post. But before that, we must decide which unique value will be used for navigating the page. File path without the file extension is a standard approach used in the community (called a slug), but we currently don't have this value available easily.

That's where Contentlayer's Computed Fields come into the picture. It lets us compute new metadata from existing content. For example, it can be used to calculate the estimated reading time from the post content.

To compute the slug field, we need to modify the contentlayer.config.js as follows -

export const Post = defineDocumentType(() => ({
  // The name of the document type
  name: "Post",
 
  // The type of the files to parse. 'md' also works
  contentType: "mdx",
 
  // The path of the mdx files, relative to contentDirPath
  filePathPattern: "posts/*.mdx",
 
  // Fields present in the frontmatter of MDX file
  fields: {
    title: { type: "string", required: true },
    published: { type: "string", required: true },
    description: { type: "string" },
    status: {
      type: "enum",
      options: ["draft", "published"],
      required: true,
    },
  },
 
  computedFields: {
    slug: {
      type: "string",
      resolve: (doc) =>
        doc._raw.sourceFileName
          // hello-world.mdx => hello-world
          .replace(/\.mdx$/, ""),
    },
  },
}))

Once the JSONs are generated, you will notice a new field, slug, added to the post -

{
  "_id": "posts/hello-world.mdx",
  "title": "Hello World",
  "slug": "hello-world",
  "body": {
    "raw": "\nHello world! This is my first post.\n",
    "code": "var Component=(()=>{var sr=Object.create;..."
  }
  // [...]
}

We can do a lot more with the computedFields. We will explore this in more detail in future posts.

Let's tie it all together

Let's build a dynamic article page with the information available. The aim is to show the nicely formatted markdown content whenever the user navigates to the <base_url>/blog/hello-world page on the blog.

Create the following folder structure in the root of the project directory -

./
└── app/
    ├── blog/
    │   ├── [slug]/
    │   │   └── page.tsx
    │   └── page.tsx
    ├── layout.tsx
    └── page.tsx

We will use the page.tsx on line 5 to show a dynamic article page and page.tsx on line 6 to display a list of all blog posts.

Blog Posts List

Open up ./blog/page.tsx in your code editor and add the following code -

import Link from "next/link"
import { allPosts, Post } from "contentlayer/generated"
 
export default function ListPage() {
  // sort posts based on 'published' date
  const posts: Post[] = allPosts.sort((a, b) => {
    return compareDesc(new Date(a.published), new Date(b.published))
  })
 
  return (
    <div>
      <h1>Blog</h1>
      {posts.map((post) => (
        <div key={post.slug}>
          <h2>
            <Link href={`/blog/${post.slug}`}>{post.title}</Link>
          </h2>
        </div>
      ))}
    </div>
  )
}

Article Page

Once we click on any entry on the blog list page, it takes us to the /blog/<slug> page, where the content of the markdown file should be shown. Let's implement this functionality now -

import { allPosts, Post } from "contentlayer/generated"
import { getMDXComponent } from "next-contentlayer/hooks"
 
type Props = {
  params: { slug: string },
}
 
export default async function Page({ params }: Props) {
  const { slug } = params
  const post: Post | undefined = allPosts.find((post) => post.slug === slug)
  if (!post) {
    return <></>
  }
 
  const MdxContent = getMDXComponent(post.body.code)
 
  return (
    <div>
      <h1>{post.title}</h1>
      <section>
        <MdxContent />
      </section>
    </div>
  )
}

Contentlayer provides the getMDXComponent hook, which can render the mdx content on the page as HTML. Under the hood, our mdx file was transformed by Contentlayer using a bundler into JSX and cached in a JSON file, which was then picked up by Next.js to statically render using React.

Conclusion

Contentlayer greatly abstracts away the intricacies of using MDX content inside a Next.js project by doing most of the heavy lifting for us. It simplifies working with unified, remark and rehype ecosystems while providing interfaces to define a schema to our relatively freeform MDX content and type-safe data structures to access the transformed content.

It is also possible to pass custom MDX components to the MdxContent to enable the re-mapping of standard markdown constructs to React components. We will dive deeper into it in future posts of the series.

Manali: Mountains and Memories

Sat, 22 Jul 2023 14:12:15 GMT

After our honeymoon got delayed for unfortunate reasons, we contemplated a short trip somewhere nearby. I had not visited the mountains until then, so I started looking for destinations in the Himalayas. Kashmir was the obvious choice, but I wanted to spend more than a few days in Kashmir which was not feasible then. Alternatives were to look for hill stations in Himachal and Uttarakhand. At last, we decided to go to Manali in Himachal Pradesh. We planned the trip for March 1, 2023, to March 4, 2023. The forecast was of rain, but still, we decided to go ahead.

Planning Stage

While planning the trip, I started looking for different options. One option was to book the tour through a tour operator in Jaipur, so the operator would take care of our commute, hotel, sightseeing and other activities. I also searched online websites like MakeMyTrip for packages. I found both these options to be expensive. Apart from that, one of my friends went to Manali last December. He also suggested not to opt for these two options. Ultimately, we decided to reach Chandigarh by train and then book a package.

To reach Chandigarh, I booked Ajmer - Chandigarh Garib Rath Express for February 28. I had never travelled in a 3AC coach on Indian railways, so this was my first experience. I really liked it. The journey was smooth. They provide everything - blankets, bedsheets, pillows, pillow covers. However, I was slightly cautious about using these items at that time due to the prevailing COVID situation. Alas, the cold forced me to ignore my suspicions.

Jaipur to Chandigarh

We reached Chandigarh early next morning at around 7 AM. One person showed us his travel agent's leaflet when we got off the train. I made a mistake here. Instead of inquiring there, I just went along with him. I blame the rain partly for this. It was showering mildly then, and I found it convenient to go with him. The guy took us to his travel agent's office, where we were explained the package. Although the tour package contained everything we read online, it had a few missing locations, which none of us noticed. Apart from this, we should've asked about hotel details. Later we found out that the hotel given to us was far away from the main centre of Manali.

When it came to negotiations, I fumbled. I should've negotiated a bit more, but I didn't, and we settled on ₹18,000 for the package. Later I was told that the same tour could've been done in even ₹14-15k.

Another thing we didn't keep in mind was to negotiate the pricing of the hotel where we were going for refreshments before leaving for Manali. Ultimately the hotel owner took ₹400 for spending just one hour in a shady room.

Road Trip to Manali

The road distance between Chandigarh and Manali is around 300 km. I usually hesitate about road travel due to motion sickness, but we had no choice. We started at around 9.30 AM from Chandigarh. After about an hour, we stopped at a roadside dhaba for breakfast.

The driver then drove without stopping until we stopped for lunch at around 2.00 PM at Mayur Restaurant in Sundar Nagar. The restaurant was good, but we were not in good shape because of motion sickness.

After having lunch, we drove for another hour before we reached the scenic Pandoh Dam. The view from here is beautiful, and the rush of air hitting your face makes you forget everything and enjoy the wonders of nature.

We spent nearly 45 minutes here before our driver reminded us that we may get late for Manali. So, with a heavy heart, we left the place and proceeded on our journey.

The River Rafting in Kullu

Afterwards, we drove for another two hours before reaching Kullu. By this time, drizzles had started, and the roads of Kullu were wholly soaked with heavenly pours. In Kullu, we had options to do Paragliding and Rafting. As the weather was not conducive, we decided to go for rafting. Despite the rains, the river was not at its full strength, and nobody was along with us for rafting. So we and our guide boarded the boat and started our adventure. We booked a one-hour rafting tour for about a 10 km stretch of river. It cost us ₹6500 for the river rafting. This includes the safety gear and guide fee. We rented out waterproof clothing for another ₹200 per person. The guide offered us to shoot a video on his GoPro for ₹1500. Though slightly expensive, we agreed because we couldn't risk taking our mobile while on the boat. Despite the exorbitant charges, it was one of a kind experience.

The guide fitted us with the safety gear - life jackets and helmets and then instructed us on handling the ride. The journey in the river started river, but soon we were hit by our first set of rapids. There was repeated turbulence beneath us, but the guide guided the boat carefully. Each time there was a strong wave, we were hit with cold water splashes from the Beas.

We spent nearly an hour in the river, and by the time we reached the endpoint, we were drenched and shivering because of the icy water of the river. We immediately ran to a bonfire nearby and started giving ourselves the heat. It was so cold that I could put my foot in the fire without hurting myself. The vendor there offered us coffee. That tiny cup of coffee cost me ₹30, but it was invaluable then. The guide dropped us at our boarding point, where our driver was waiting. After payment and getting the videos on our phones, we started our journey again. After this, we were to stop only at our hotel.

The Hotel

Our hotel, The Snow Mount, was so remote that a tourist couldn't reach it without a car. We were very disappointed with our haphazardness while driving to this place. By the time we got there, it was almost 8.30 PM at night. We were starving, so we decided to order simple food. We were exhausted due to car travel, so after finishing dinner, we decided to call it a day.

Solang Valley

The following day, I woke up early and as soon as I went outside to check the balcony, all my frustration disappeared. The view was breathtakingly beautiful. We quickly got ready and came downstairs for breakfast. After having breakfast, our driver suggested that instead of going to the city sightseeing, we first go to Solang Valley because the weather forecast was gloomy for the next day. We agreed and proceeded to visit the Solang Valley. On the way, we stopped to book activities at Solang Valley. We booked zipline, mountain bike, Yak ride and tube ride. Looking back, I think we could've done with zipline and Yak rides only as well. All these activities cost us ₹5000. The snow dresses cost another ₹500.

Ziplining

We had never done ziplining, so it was a unique experience. Although the height of our zipline was not much, as a first experience, I am satisfied with it.

First view of the valley

After this, we went straight to the Solang Valley. This was my first view of the snow in real life, and I was totally awestruck by the vastness of it. We went in the low season, so a lot of snow had melted, but still, I could see the ocean of snow in front of me. What would be the scene after the heavy snowfalls we hear about in the news?

Mountain Bike

When we finally reached the valley, the driver dropped us at the parking, from where the tour operator's mountain bike picked us up. The terrain was rocky, and we needed to go uphill. The ride was full of ups and downs, and we enjoyed it thoroughly. Was it a must-have? I don't think so. It is a good experience, but if you don't opt for it, you won't miss it. Later, I also got a chance to drive it.

Yak Ride

After this, we were taken to Yak Ride. I saw a Yak for the first time and was instantly mesmerised by this animal. We were allowed to spend about 15 minutes on the animal.

Tube Ride

After the Yak ride, we went for a tube ride. I don't know what is the point of this activity. People were being slid from the top of a cliff in a rubber tube on a snow trail. It took not more than 10 seconds from top to bottom, and that's it. You are done. This is the height of looting people. I fell for the same and immediately regretted it.

The Anjani Mahadev Temple

After this, our activities were over, and we were free to roam around on our own. Initially, we thought that was it. But then people were going further uphill. So we decided to go as well. We had to hire mules to reach the top as the slope was treacherously steep.

Near our drop point, a stream of water comes down from the mountains. It is quite a picturesque location. We also collected some pebbles from here while on our way back later.

So, on the hilltop, there is the Anjani Mahadev Temple. A few sadhus live and worship the temple deities there. We went inside the temple, prayed and came outside. There was a great rush of people here. Hence, the sadhus repeatedly got angry about people's negligence of the temple decorum.

Going even further, a waterfall on a hilltop continuously showers a Shiva Linga inside a well. At that time, the way to the hill was totally covered in snow, so it was pretty dangerous to climb to the top. But we saw a few Malayali (I keep finding them even now!) boys doing it, and with their help, we decided to do it as well. They had managed to get a few sticks for support. We borrowed two wooden sticks from them and started our upward trek. Once we reached the top, it was time to slide down. We climbed to the top to do this. However, it was pretty dangerous. One wrong step and you will hurt yourself badly. We reached the bottom, however, with some bruises on my palms.

When we reached down, we saw a massive snow plaque with the date 2.3.23 written on it. We decided to talk to the vendor and get our photos clicked. It was a paid service, but totally worth it because we completed three months of our wedding on this day. And clicking pictures here made this day memorable.

After this, we hired another mule and began our descent. Once we reached the parking spot, we undressed ourselves and decided to bid farewell to Solang Valley. It was a day of unique experiences and many firsts.

Nehru Kund

On our way to Mall Road, we stopped at Nehru Kund. As per our driver, it is a freshwater stream whose source is unknown. It was not crowded at all. The water was coming from a plastic pipe. I think this place doesn't need more than a passing glance.

Van Vihar

After this, our driver dropped us at Mall Road to roam and asked us to be ready for pickup by 8.30 PM. So, we had around 3-4 hours to wander around. So we decided to take a round of Van Vihar. It is on the main Mall Road, so it wasn't difficult to find. This is a park adorned with sky-touching deodar trees. It is good enough to spend 1-2 hours amidst the tree away from the noise of the Mall road. After this, we shopped and had dinner at a Punjabi dhaba on Mall Road. The food was average.

City Sightseeing

We planned a tour of famous tourist attractions in and around Manali the following day. We visited Hadimba Devi Temple, Clubhouse, Manu Temple, and Tibetan Buddhist Monastery. People also go to Vashisth Bath, Nagar Castle and Rohtang Pass, but our tour operator didn't include these in our plan. We were careless while booking our package and had to pay the price during our entire tour.

Hidimba Devi Temple

First, we went to Hidimba Devi Temple, a temple dedicated to Hidimba, the wife of the Bhima, the second of the Pandava brothers from Mahabharata. It is a 500-year-old temple built at the foot of the Himalayas.

After getting a darshan of the Devi, we strolled in the nearby Dhungiri Van Vihar. It is a lovely little forest covered with Cedar trees. It also has a tiny water pond.

We also visited Ghatotkacha Temple nearby. Locals have also built a temple dedicated to the grandson of Hidimba, Barbarika, at around one km distance. We also decided to visit that temple.

Clubhouse

After the Hidimba temple, we went to the City Clubhouse. There is hardly anything to do at this place. The whole place is a sham.

One can go near the river to take pictures or just sit on the stones in the middle of the river. On the river bank, ziplining activities are available, but we didn't opt for them. There are some souvenir shops, but I suggest not buying from them. Instead, buy from Mall Road. You will have a good chance of bargaining there.

Manu Temple

Next, we went to Manu Temple, which is about 2 km away from the Clubhouse. The journey was uphill, and soon we regretted our decision to walk. Alas, we managed to reach the temple. Even here, there is nothing to see. According to Hindu scriptures, it is said to be the only temple dedicated to Manu, the first human.

After this, we came back to Mall Road. The driver dropped us there just like the last day and asked us to call him when we were ready to return to the hotel.

Tibetan Buddhist Monastery

After arriving at Mall Road, we went to the Tibetan Buddhist Monastery, which is very near the Van Vihar. When we reached there, it was very crowded, with waves of people coming in and out of the monastery as if it were a picnic spot. But, how can I complain when I also went there for sightseeing. It is a nicely maintained place. However, everything is written in what I suspect is the Tibetan language, so even if I wanted to understand the context behind the statues and iconography, I couldn't.

We ate something interesting here - Laping, a Tibetan speciality filled with noodles and Paneer, and served with red chilli chutney and soy sauce. This was very spicy, and I was left with teary eyes after eating this. To save myself, I had to run to get Gulabjamuns sold by street vendors near Van Vihar. Apart from that, I really loved this dish. We also interacted with the old man who served in the Indian army as a sepoy 30 years back and now sells these lappings to pass his time.

Shopping

After visiting the monastery, we were done with seeing all the attractions mentioned in our itinerary. So, we decided to do some shopping. Given the experience of the past two days, we were determined to only buy something with good haggling. We spent nearly 2 hours roaming the area and purchasing souvenirs for friends and family. Later in the night, we went to La Pino'z Pizza on Mall Road for a quick dinner.

After dinner, we were roaming on Mall Road, and a random man came near me and asked me - “Sir, joint loge kya?”. I was stunned and immediately said no and left the scene. I hope he came to me randomly and not because of my appearance 🤣.

After this, it started drizzling slowly, so we called the driver and returned to the hotel.

Back to Chandigarh

We decided to leave early the next day as our train to Jaipur was scheduled for 7.30 PM. Earlier, we had planned to go Paragliding in Kullu on our way back to Chandigarh. However, we were so exhausted that we decided to do in future.

While travelling, we stopped at Shri Ram Shawls in Kullu to buy shawls for our family. Our driver took us here, saying this is the best outlet to buy Himachali shawls in Kullu. We liked the quality of the products there, but there was no bargaining allowed. We will only learn about the quality in the next winter season. We also stopped at a roadside vendor to buy some local dried fruit items, which tasted good.

After this, we didn't stop at any other place apart from for lunch at Annapurna Bhojnalaya in Sundar Nagar. However, due to unforeseen circumstances, we still missed our train. First, we had a minor accident near Mandi. Then one of the tyres of our car got punctured near the Haryana border. Once the driver got it fixed, we were stuck in a long traffic jam. All these incidents combined, and we missed our train by one hour. So the next thing I could do was to immediately book bus tickets for both of us. We booked sleeper class tickets in an AC bus as we both were exhausted and wanted to sleep. The driver dropped us at the bus pickup point in Chandigarh and bid us goodbye. After we boarded the bus, the next thing I saw was the Narayan Singh Circle in Jaipur.

Lessons

This was my first trip with my wife and the first trip where no friends were involved. It taught me the importance of bargaining on any trip. I should've tried to bargain a lot more with the tour agent. I lost a significant ₹2-3k there. Later I was cautious, but the street vendors tried to dupe us. But thankfully, we managed to negotiate the prices a bit on almost all occasions.

Also, I needed to be more careful while discussing the tour details with the agent. If I had paid more attention, I could've persuaded him to add a few destinations to the sightseeing package and extracted more value for my money.

Conclusion

This was my first proper trip with my wife. Our honeymoon got delayed, but we didn't delay too much either. This was a short trip to a place I always wanted to visit. I have always been attracted to the grandeur of the Himalayas, and seeing them from so near was a fantastic experience.

How I built a blog with Next.js

Fri, 14 Jul 2023 14:55:24 GMT

Hugo, the fastest static site generator of the world, served me well for almost 7 years. I made several iterations to my blog in these seven years. However, in these years, the web has advanced by leaps and bounds, and my aspirations have grown by an equal magnitude, if not more. I began to desire a lot more from my blog. I wanted a new design using the latest tools and real-time data fetching using APIs. Above everything else, my zeal to keep learning kept me dissatisfied with Hugo. Hugo made some improvements to incorporate JS, but I neither had time to understand these nor was I convinced about it. Perhaps, I knew Hugo so well after working on it for so long, that the idea of transitioning away from it felt intimidating.

At last, I decided to start looking away from Hugo in 2021. I was fixated on migrating to a JavaScript-based SSG for almost two years. I finally tried my hands on Gatsby in 2022, but I failed spectacularly — perhaps because I was in my courtship period then, I couldn't devote enough time to learn the intricacies of a new framework.

After that, I shelved the idea due to lack of time. However, the complexity of writing in markdown kept preventing me from writing continuously, which meant there was almost zero activity on my blog.

Why now?

About two months ago, while casually browsing Twitter, I came across a tweet mentioning the use of Notion as a CMS for a Next.js blog, all thanks to the newly released Notion API. As soon as I read the post, I knew I had to try it. The reason was simple - it would help me solve my most prominent problem of the past few years - the separation of content from the code. I was long looking for a way to solve this issue. Notion's editor is intuitive, full of features, easy to use and available on all platforms I use regularly.

Roadblocks

It was time to start brushing my web development skills again. React is in trend currently, and Next.js is a popular framework built upon it. I had also heard about TailwindCSS and how it is used to create almost every website nowadays. So I decided to use these two to build my website.

However, it was more complicated than I thought it would be. My lack of knowledge of TypeScript and TailwindCSS meant I struggled to get basic functionality working for almost one month. Fetching and integrating Notion data using its API proved even more tedious. The state of Notion API, the performance issues, and the community support are not in ideal shape for an absolute JS beginner like me. I found Contentlayer, which is a translation layer to convert the content into JSON format suitable for use in code. It has a plugin to convert Notion data to JSON. However, this plugin is in the alpha stage, and has several limitations.

As I struggled with adding Notion support, I made a tough decision. I decided to drop the idea of using Notion as CMS for the time being and use the MDX for storing the content while simultaneously honing my design and JS skills and return to Notion once I am more confident about my skills. To reduce the complexity, I have decided not to use any custom React component in my MDX files. I will still write in Notion and once the article is ready, I will copy-paste it to an MDX file and commit the file to Git. A bit of manual setup is involved here, but I am okay with it. This setup would've worked with my Hugo blog, too, but over time, I had integrated so many custom shortcodes that the markdown files no longer looked like markdown. This time, I am intentionally sticking to the standard markdown syntax to simplify any future migration.

Base Design

I found it challenging to adapt TailwindCSS, partly because I was returning to CSS after a long time and partly because I have had problems understanding technologies that tend to abstract away the complexities of underlying concepts. I still remember my struggles with understanding Python purely because I got so habitual to coding in the C language that the built-in methods provided by the Python language confused me.

As I struggled with Tailwind and coding in raw CSS was not a good option with Next.js, I decided to look for a portfolio that could act as a base for my changes. The portfolio of Emma Campbell was the right fit for my requirements. It was built with Next.js App Router and TailwindCSS. It had all the features I required (except a few, like Table of Contents). Above all, I could understand the codebase, so it was easy to dive in and make changes to the design.

Features

Some functionality was already present in Emma's codebase -

Contentlayer integration
Views Counter using Kysely and Planetscale
Code highlighting

I added/improved several other features to the codebase -

MathJax support
RSS feed and sitemap generation
Dark mode support
Giscus comments
Umami and Cloudflare analytics
Simplified code highlighting implementation
OpenGraph image generation

I plan to add following features to this blog in future -

Table of contents
Image component improvements
1. Caption support
2. Gallery and “Click to Expand” feature
Tooltips
404/500 pages
Change OG image generation implementation
Add a dark mode toggle button
Full text search on blog page
Move to Redis to reduce latencies
Proper error handling in components
IndieWeb

and many more …

Inspirations

This blog was only possible with many amazing people open-sourcing their blogs. I took inspiration from their design and sometimes picked up their code. Here is a brief and incomplete list of blogs that influenced me a lot - Emma Campbell, Lee Robinson, Delba de Oliveira, Anish De, and Josh W Comeau

Coming Next

I plan to write separate articles explaining how I implemented every feature mentioned above. Stay tuned!

2022 - The Year of Plentiful

Sat, 31 Dec 2022 13:25:03 GMT

2022 is about to end. It is that time when I say goodbye to one more year in my life and welcome a new one. 2022 was a hectic year for me. Many of my decisions this year will have rippling effects for years to come. But there was also a good realization that I must take action; sitting idle won't help.

As always, I have divided this review into those areas of my life where I have been putting more focus throughout the year.

Health

Fitness

I wish I had put more focus on my fitness this year. My sedentary lifestyle may put me at many health risks in years to come. My activity levels have been abysmally poor for most of the year. A big reason behind this was my busyness in other parts of my life. 2022 was an amalgamation of stress, fear, anxiety, surprises, gratitude, and happiness. Fitness took a back seat due to the quick turnaround of events throughout the year. Early in the year, I continued with Yoga. However, I was a lot more irregular this year because I wanted to gain muscles while keeping my weight constant. So, I decided to join a gym in the second half of the year to focus on muscle building. However, due to the stress of wedding preparations, I couldn't find the motivation to continue beyond two months. There was one positive, though - whenever I committed to working out, I could notice a boost of confidence spurting inside me. Unfortunately, I found it challenging to maintain this motivation for long.

I didn't have to suffer from any significant illness this year. I occasionally suffered from slight back pain, which can be attributed to my poor posture. Doing Yoga keeps my back in shape, but whenever I stop doing Yoga, I start feeling the pinch again in one or two months. This factor alone makes me realize the importance of keeping my body active.

Sleep

I managed to sleep around seven hours each day throughout the year. However, this data doesn't include my afternoon naps, so my daily sleep duration is undoubtedly higher than 7 hours. Sleeping in the afternoons has become a habit that I never intended to develop. But now, it has become a permanent fixture in my life. In 2023, I plan to keep my tendency to sleep in the afternoon in control, but that will require sleeping enough at night, without which all my efforts will remain useless.

I slept the least in June, which is surprising. I was expecting it to be in July, August, and September when I was in the midst of my courtship period.

I would have liked to put a graph showing my sleep and wake-up times as well, but my graph plotting skills require some polishing, so for now, I will stop here. Someday, I would like to analyze that data as well.

Career

Netskope

Joining Netskope was a good decision for my career. While I learned a lot at Cisco, my experience at Netskope has been different. This year, after seeing the turbulent market conditions, I kept on wondering if joining Netskope was a financially prudent decision. If I keep this fact aside, my experience at Netskope has been good.

Learning

I have not been able to keep up with emerging technologies in the last two-three years. While my basics are solid, I should invest some time in learning new trends to stay relevant and employable in the future. The tech industry can be fierce at times. With all the turmoil in the world, it becomes even more important to keep myself updated. For the last four years, I have been lacking in learning new things (irrespective of their connection to my work). That may have put me behind my peers by a few years. Whenever I see my peers racing ahead of me, I feel disappointed. I know that becoming part of this rat race will only give me grief, but as a human, I am not immune to these emotions. However, I am very well aware that taking action is the only way to come back on track.

Future

For the first time in the last three years, I feel clarity in my thoughts about my future direction, which may be a good sign. Going into 2023, this clarity can prove miraculous for my career and personal life. I hope it is the case.

Writing

The Problem

Writing regularly was one of my primary goals this year. For the last couple of years, I have been struggling with some mental block when it comes to writing. It turns out that the problem was the lack of content. I either did not have anything to write about, or needed to try harder to gather the material, or procrastinated. I wrote less this year too, but I could get into the flow whenever I wanted to.

Journaling

Blogging was not a priority this year. However, journaling has proven to be a wonderful experience for me this year. More so, it gave me a window into my thought process when making the most critical decision of my life - whether to marry or not. I wrote extensively around that time about what I was thinking, what I liked in her, and what I didn't. When I reread those entries, it immediately brings a smile to my face. Such has been the impact of journaling on me. I plan to write many more journal entries next year. I hope these new entries will also bring a smile to my face in the distant future.

Hobby

I have a long-term wish to start a newsletter and grow my audience, but I wonder if I am ready to put in the effort required for this task at this moment. Also, I need to be more confident about whether I have good content/topic to write about. Since I still hesitate to share my blog posts on social media, starting and growing a newsletter will be an uphill task.

Getting committed this year meant that I finally had some inspiration to engage with creative writing. And I thoroughly enjoyed writing for someone else. In the last seven years since I started writing, I have never tried to write in Hindi. But after doing it this year, I realized that I enjoy writing in Hindi. I plan to continue this (even if sporadically) in 2023.

Reading

The failure

I have a feeling of disappointment while writing this section. I decided to read 24 books this year. I am ending it without reading a single book. Throughout the year, I could not make up my mind about reading. A significant reason behind this was the double thoughts in my mind about the objective for which I was reading and whether I could retain whatever I was reading. I found myself struggling to come to terms with the fact that no matter what I do, I will never be able to recall everything I read. This realization was the root cause behind my disillusion with reading. Unfortunately, I still need to find a definite answer to this dilemma. However, I can still restart my reading journey with fiction books, as I don't see the need to remember such books.

But…

Having said this, it is not that I didn't read anything at all. I managed to read the first three modules of Zerodha's varsity content. These modules are such a valuable source to learn about the markets. I thoroughly enjoyed reading them. Unfortunately, at least this year, I didn't use the knowledge gathered. To start with the stock market again, I may have to reread the modules to recall the concepts (this is one example of the issues I mentioned in the previous paragraph).

Media Consumption

Bingeing

One side effect of not reading enough books was that my mind tried to find other avenues to pass the time. In the first half of the year, I took refuge in movies and TV series to relieve myself from day-to-day life. I watched 22 films and three web series in the year's first half. Before getting married, I watched only two movies between July and November. After getting married, in December, I watched five movies.

Name	Type	Month	Rating
Loki	Show	January	7
Eternals	Movie	January	6
Pushpa: The Rise - Part 1	Movie	January	7
What If…	Show	January	8
Kingsman: The Golden Circle	Movie	February	8
The King's Man	Movie	February	6
Mortal Kombat	Movie	February	4
Rudra	Show	March	7
Spider-Man: No Way Home	Movie	March	8
Bell Bottom	Movie	March	5
Wanted	Movie	March	4
Housefull 4	Movie	March	2
Dasvi	Movie	April	7
National Treasure	Movie	April	7
Badhaai Do	Movie	April	6
National Treasure: Book of Secrets	Movie	April	6
The Batman	Movie	April	9
Pirates of the Caribbean: The Curse of the Black Pearl	Movie	May	8
Pirates of the Caribbean: Dead Man's Chest	Movie	May	7
The Lost City	Movie	May	3
The Chronicles of Narnia: The Lion, the Witch and the Wardrobe	Movie	May	8
The Chronicles of Narnia: Prince Caspian	Movie	May	8
The Chronicles of Narnia: The Voyage of the Dawn Treader	Movie	May	9
K.G.F: Chapter 2	Movie	May	7
Doctor Strange in the Multiverse of Madness	Movie	June	8
Brahmāstra Part One: Shiva	Movie	September	7
Thor: Love and Thunder	Movie	October	7
Doctor G	Movie	December	7
Cirkus	Movie	December	5
Govinda Naam Mera	Movie	December	6
Encanto	Movie	December	7
The Dictator	Movie	December	6

I was in control of my social media usage in the first half of the year, but Instagram caught me off guard after that. I have never been a fan of Instagram, but this year, circumstances forced me to get into the rabbit hole. Currently, I have managed to control my Instagram usage. Still, I need to keep my eyes open to avoid any addiction.

Phone

I don't have my phone usage data, but I am sure it has skyrocketed this year. Since July, I have managed to use my phone for more than 6 hours every day on a routine basis. I must figure out how to reduce this usage and focus on more important things.

Family

Back to Office

Since the beginning of the pandemic, I have been staying with my family. It has been three years, and with each passing day, I feel uncomfortable even thinking about moving back to Bengaluru. With the feeling of belongingness combined with the mess that Bengaluru is in, I find myself resisting moving back to Bengaluru, at least in the near future. My employer supports remote work, which gives me hope that I can spend a few more months here in Jaipur. Staying in Jaipur has obvious monetary benefits, but I see long-term hindrances in career growth. I may have to find a middle path to avoid stagnating in my career and satisfy my emotional cravings towards my family.

Fam

On the family front, this year has been incredible. Several long-term wishes of my family got fulfilled this year. My mother came back home after a successful recovery from a severe ailment. We shifted to a new house. I got married. My brother got a government job. What else could I have asked for?

Vishi

As the latest entrant into my family, she has been a welcome addition. When 2022 started, the idea of getting married was completely absent from my mind. There were other more important stuff in my life that required immediate attention. But life had its plans. And I am glad that life had its way. Though I was not rooting to get married, I found a person with whom my thoughts match. I have not been a people-friendly person in my personal life, so it will take some time to adjust to another person's presence. Still, the way we have spent the last five months together, this will be a fun journey together.

What in 2023?

My experience says I could improve following up with my new year's resolutions. However, that is not going to stop me from resoluting some more goals this year as well. Even if I don't achieve any of these, they will linger around in the back of my mind and remind me to take at least one step toward them.

Fitness

I don't know of any method to quantify my fitness progress, but I aim to keep my weight under 65 kgs while transferring some of my weight from fat to muscles.

Reading

I have decided to read at least 24 books (2 books per month) this year. A combination of fiction, non-fiction, Hindi, and technical books would be ideal and help me avoid the sudden lack of interest. You can track my progress here.

Writing

Some journal entries, FutureMe letters, and a few blog posts should be enough for this year. I am writing more than ever privately, but as a side effect, public writing has taken a back seat for now. I will write on this blog whenever I have something important to share.

I enjoyed journaling in 2022, so this is also high on my priority list for 2023. I intend to write only sometimes, but doing it once in a while - on important events of my life - should be doable.

Travel

Now that I have a partner-in-crime, I plan to travel extensively (while keeping financial and professional commitments in mind). The resurge of COVID is threatening to spoil one of my trips early next year, so I also need to keep an eye on that. However, I will travel as and when possible. Expect some travelogue posts on this blog in the near future.

2022 was a good year for me. I achieved many goals this year. After a long time, I have some clarity in my mind about my future course of action. I hope to execute some (or all) of my plans in 2023.

This year taught me a vital lesson - every time you plan something, there will be a lot of apprehensions. All it takes to proceed further is to take one step. Everything else automatically falls into place.

Thank you, 2022, for this critical lesson. Adios, I will never forget you for bringing so many new memories into my life.

And welcome to you, 2023!!!

I Got Engaged 💍

Sat, 06 Aug 2022 12:05:24 GMT

Life sometimes takes such unexpected turns that we humans can only stand still and wonder. That's what happened to me. Just two months back, I was urging my father to stop talking to me about my marriage plans, and here I am today, announcing my engagement. It is how the sequence of events has been in my life in the last two months. However, I am not at all troubled with all these sudden turns and twists.

Coming to the topic of the discussion, the name of my fiancée is Vishakha. Our roka-cum-engagement ceremony took place on 30th of July. We exchanged rings before a small family gathering and had our first pooja together. Now, we are officially in our courtship period. We both belong to slightly different educational backgrounds and academic interests. However, that doesn't seem to be a barrier between us, and we share many views about life, family, society, and other things in general.

I have been timid in talking to strangers, but right from our first meeting, I struck a strange connection with her. Our first meeting lasted for around one and a half hours, which astonished our families, given that she and I don't talk much, even among family members. Since then, we have spoken for uncountable hours, discussing our priorities, ambitions, goals, thoughts about life, likes, dislikes, etc.

It may be initial excitement or infatuation, but I can feel the emotions of care and love blossoming in me. These are some unfamiliar emotions for me. I have never attached to any human being other than my parents. Now having the same feelings for a stranger is difficult to digest. I don't know if love can blossom so quickly. But if it does, then I am definitely in love. And if it doesn't, it will happen soon.

When I broke the news on Instagram, many people asked me one question - Why so early? While there were no particular reasons behind my decision, in retrospect, I think this was an excellent decision to take at this juncture of my life.

I am currently stagnating in my life and career. I am going through that phase where I don't see any yield of my work on both personal and professional fronts. Although I am confident that I will reap the results of this work in the future, the present appears to be the best time to welcome another person into my life. I hope her entry into my life and the resultant influence will help me come out of this slump phase. I must mention that this is a double-edged sword, and I am taking a huge risk here. But that is something only time will tell.
My parents would have started forcing me to get married in the next year or two. I preferred not to juggle between Bengaluru (or elsewhere) and Jaipur, hoping to find a match. I have seen my cousins struggling with that.
I had planned to go for a corporate-sponsored graduate program. But then I left Cisco, and since Netskope doesn't have any such program; I had to put this plan on hold for a couple of years. So, I am relatively free at this point in my life. I believe there couldn't have been a better time to hook up than this.
After the pandemic began, I came back to my hometown. Since then, my life has become highly monotonous, and I was looking for some respite from it. I was feeling the need for another person in my life who could push me out of my comfort zone while also respecting my boundaries.
Perhaps I was also looking for lady luck.

Marrying means leaving your bachelorhood behind, and that has its downsides.

Time spent on your family means you can't spend time on your career and other outside-of-work activities. But I think, at some point, I had to find a balance anyway.
The responsibility to maintain a family is not an easy one. It comes with its challenges. Bachelorhood is relatively easy; you are on your own. But in married life, someone else is looking up to you for constant support, dedication, and love. I wonder how long it will take me to adjust to this.
Your goals and ambitions are not only yours anymore. They get shared with your spouse. And if the spouse is not supportive, life can take a downturn quickly.
You can't just think about yourself selfishly. You need to consider the effect of every decision of yours on your spouse as well. It can get clumsy sometimes.

As usual, I have spoiled this post by talking about ups and downs 🤣. My habit of weighing everything in pros and cons will not die so quickly 🤓. But I want to clarify one thing: I am thrilled and content with my decision.

I am looking forward to this alliance and spending more time getting to know Vishi 💗. I hope I will be a worthy partner to her.

ISRO interview experience and takeaways

Sat, 29 Jan 2022 09:01:58 GMT

Note: This post was originally written on Jan 29, 2022 but I never posted it. Posting it now keeping the original authoring date, but with commit date of Aug 05, 2022.

This post was on my bucket list for almost one year, but I kept ignoring it for one or another reason. Anyway, I wanted to write this post to add my experience to several other interview experiences to help others. Another intention was to document this experience for my own consumption in the future to relive one of the greatest moments in my life and career without worrying about diminishing memory.

I had decided to take the ISRO exam as part of my GATE preparation strategy. The paper was apparently very lengthy and came with time-consuming calculations. With some difficulty, I could solve a fair number of questions. Then, I forgot about this and started focusing on my GATE preparation again. The GATE journey is also memorable, which I may describe in a future post. While I was waiting for results for both ISRO and GATE, the pandemic hit the country, and I made a quick jump back to my hometown. The pandemic also delayed the ISRO results. But when it was announced, it was a surprise for me. I was selected for further interview rounds. Then, a long wait ensued. I had to wait till February 2021 to get an interview call. I attended the interview for the post of 'Scientist' in ISRO on March 4, 2021, in New Delhi.

I had been part of very few interviews in my life until ISRO's interview. I could count them on fingers -

SSL admin interview at NITC (which I rocked :P)
The Directi interview (where I reached till last round), and
The Cisco interview (I rocked this one too).

I haven't been part of any other interview in my life, so far. So, I am notably less experienced in giving interviews. But, unlike other fields in my life, I have never felt nervous while giving interviews. I am not yet sure of the exact reasons behind this confidence. Perhaps, I am not an introvert as I think I am.

Anyway, the decision to attend or not attend the interview was tough. There were many reasons -

What if I get selected for the interviews! The pressure to join ISRO will be enormous.
What if I get COVID during the journey or stay in New Delhi!
If I am not interested in joining ISRO, why should I waste money on travel and accommodation!

My parents didn't know that I had taken the ISRO exam. So, it was a pleasant surprise for them. I decided to attend the interviews because my parents looked ecstatic about this opportunity. My brother accompanied me to Delhi; we stayed in a nice hotel, ate good food, and visited some excellent locations in Delhi together.

Coming to the primary purpose of this post, how was my interview experience? Well, it was a mixed bag. I wasn't satisfied with some of my answers, but I could feel that the interview panel liked my answers. Like all other interviews I have given, this interview revolved mostly around Linux, Networking, OS, and Security. One good thing about ISRO interviews is that ask your preferred subjects before starting the discussion. I said that my favorite subjects are OS, Security, and Networking. They did ask me why I don't like DBMS. I told them that I had never worked with DBs until now, so I am not very familiar with its practical aspects. After that, they asked me questions from my preferred domains only. As expected, I had absolutely no issues answering most of the questions accurately. There were some tricky questions where they tried to stress-test me, but I was able to deflect them.

Although I had to be physically present at the ISRO center, the actual interview happened virtually on Cisco WebEx (!).

Below is an inaccurate version of the converstaion -

Interview Panel (IP): Please introduce yourself.

Me: Introduced myself. The panel specifically asked about my graduation details and final project details.

IP: So, you work in Cisco. What are you doing there?

Me: Explained my work in Snort and FTD team

IP: What purpose does FTD serve?

Me: Explained in brief, as I had just joined the FTD team at that time.

IP: We are using FTDs in ISRO. It is giving us too many issues. How do we fix it?

Me: Suggested some solutions, including Hardware and Software ones. They tried to stress-test me on this. Thankfully I didn't lose my patience and kept giving them some answers.

IP: What happens behind the scenes when you type a web address in the browser?

Me: Explained

IP: How does the router know where to forward the packet?

Me: I fumbled a bit on this one because I did make a mistake in the previous question. Anyway answered it correctly in the end.

IP: What is page fault and context switch? Does a page fault require a context switch?

Me: Answered. Don't remember the answer now. It is very clearly explained in OSTEP.

IP: How is kernel mode switched?

Me: Explained

IP: What is a trap? What does the trap handler do? How is it invoked?

Me: Explained

IP: Explain Linux File Permissions? How to interpret directory 555 and file 777 permissions?

Me: Explained

IP: Do you know about the security of OS?

Me: Explained some concepts. I mentioned Linux Hardening, then they asked many questions on the overall idea of hardening of the OS.

IP: How to modify a Kernel to work with resource-constrained devices?

Me: I explained some modifications, like removing unnecessary drivers and blobs used for generic purposes, etc.

IP: What is a TCP wrapper? Do you know about Netstat?

Me: I didn't know about TCP wrappers, so I said I don't know. Explained Netstat briefly.

IP: How can we improve the security of an OS?

Me: I explained some ways to secure OS connected to a network by installing some tools and firewalls, like Fail2Ban, FTDs, UFW, etc. They were expecting IPTables, so they asked about it next.

IP: What are IPTables? Do you know about chains in IPTables?

Me: I explained IPTables, but couldn't answer the concept of chains.

IP: Explain some common web attacks?

Me: I explained OWASP Top 10 and a few attacks like SQL Injection, XSS, etc.

IP: Which attacks are possible in an OS?

Me: Explained buffer overflow, heap overflow, stack smashing, etc.

In the end, they asked some general questions about Linux Kernel, which I don't remember now. Then they concluded the interview and asked me to leave. The interview took around 30-45 minutes. The discussion entirely revolved around the systems side of things, which worked in my favor.

I analyzed my performance after the interview. There were some things that I would like to fix in the future.

While preparing for GATE, I didn't focus on making notes. Having short notes helps at the last moment. I suffered because I had to go through the entire syllabus again, even for a quick revision.
I have improved my communication skills significantly, but they are still not up to the mark. My inherently introverted nature keeps pulling me back. During this interview, I fumbled quite a bit. It may be that seeing myself on camera on screen made me self-conscious. Still, I need to work on this aspect of my soft skills because I am sure that I will need to use communication skills a lot in the coming future.
My knowledge of Linux, Networking, and OS is invaluable. I need to make sure that I don't neglect them in the future and keep myself up to date with the latest trends in the field.
I severely lack focus and motivation. Even during the interview, I faced issues with focusing on the questions. My mind keeps on having distracting thoughts. I need to find some ways to improve my focus significantly. Otherwise, my mind will lose its capabilities even faster with growing age.
My understanding of Cisco products is severely limited. I spent three years in Cisco. Still, I don't know much about any Cisco product lines completely. I don't even know about all Cisco's Security unit's products, even after spending my entire 3 years there. This is a difficult situation. Unfortunately, now that I have left Cisco, there is no way/reason to get that understanding.

The results came in September 2021, and to my surprise, I was selected for the post with AIR 30. My final marks in the written exam were 110/216 (some questions were canceled), and I scored 83.18/100 in interviews. However, I decided not to join ISRO because the center assigned to me didn't align with my interests and other reasons that I mentioned in a separate post. My parents were delighted with my achievement; however, they supported my decision not to join ISRO.

August 2018 to February 2020 was a very frustrating yet fruitful phase of my life. The knowledge I acquired during this time helped me overcome my self-doubt due to my failures in academics in NITC. Getting selected for ISRO made the entire process memorable in the end. I will always cherish this achievement of mine.

Why I decided to not join ISRO?

Wed, 26 Jan 2022 15:19:58 GMT

Note: This post was originally written on Jan 26, 2022 but I never posted it. Posting it now keeping the original authoring date, but with commit date of Aug 05, 2022.

Sometimes, life presents you with some choices that it becomes tricky to choose any one of them. Last year, my turn came when I was selected for a prestigious job at Indian Space Research Organization (ISRO). I was chosen for the post of Scientist in the organization. The pay was top notch with the pay grade equivalent to that of new bureaucrats. The pride that comes with the service to the nation was also associated with the job. I could have enjoyed my life without any worry about layoffs, pay cuts, workload, always having to learn new things, etc. But I decided to leave the offer and continue with what I was doing.

Why did I do that? Was I mad? It has been almost one year since my final interview and nearly 3 months since I declined the offer. In this post, I intend to publicize my reasons for not joining ISRO. You may or may not agree with a lot of these reasons (even I doubt some of my decisions), but don't hesitate to voice your opinion. Perhaps, it will give me another perspective to think about similar situations in the future.

The job profile and assigned ISRO center

I studied Computer Science during my undergraduate and have been working in the software industry since then. I know I can learn any new field (aerospace engineering, in this case) if I want to. Still, it was not a financially lucrative choice, and also it was not in alignment with my interests.

I was assigned to NRSC, Hyderabad, which would be my base for the rest of my career. NRSC stands for National Remote Sensing Center. As I understand it, the NRSC primarily works on analyzing the images and media received from various ISRO satellites. I have never been an enthusiast of the field of computer vision and image processing (perhaps because of the maths associated with it). So I wasn't very keen on doing something which does not align with my current interests. Although, one can say that you start loving anything if given enough time. But I wasn't very sure about taking chances.

Of course, I queried about the possibilities of transfer to some other center. I was told that the chances of a transfer are slim unless there are exceptional circumstances or a mutual transfer.

The location

As I mentioned earlier, if I had accepted the offer, I would have been bound to Hyderabad for the rest of my career. This was totally unacceptable to me. Many things have changed globally since the pandemic started, and the work location flexibility is one of the positive changes. Earlier, it was challenging to work from home, but now, it is/will be accepted as an industry norm. ISRO, being a government organization and from a national security perspective, will not allow its employees to work from home, rightly so. At present, I would perhaps be okay with the idea of exploring the world and staying away from the family. Still, I would like to settle down at some familiar place in the longer run. Joining ISRO would have stolen that possibility from me ultimately, if not immediately.

I have been staying at my home for the last 2 years due to COVID lockdowns. This has affected my psyche at a deep level. I no longer wish to stay away from my parents. The detachment caused by my 7 year-long explorations ended with COVID lockdown. It will be tough for me to stay away from them for any long duration now. Air travel has reduced the distances drastically, but the possibility of a WFH-friendly future has spoiled me. ISRO is unlikely to provide this kind of option in the foreseeable future.

Money

Money always plays a part in every decision a working adult takes. While the starting salary given by ISRO is comparable to private sector companies, the yearly increments are peanuts. One gets 18% conditional increment + possible promotions every four years, which looks terrible compared to MNCs. Although joining ISRO comes with certain other perks which will definitely match or even outshine the overall CTC offered by MNCs in some cases.

Suppose I had decided to join ISRO in an alternate universe. In that case, I could have given you a lot of reasons why money should not be a factor in deciding whether to take the job or not. So, I will stop discussing the importance of money in a job offer here.

Not my kind of government job

Government jobs are lucrative in India. They pay decent, have excellent perks, give life-long job security, and calm and peaceful life on most days. However, you can divide the government jobs into three categories -

Those with power, status, money, meaningful work, and hectic life
Those with meaningful work, money, and peaceful life but no power and status
Those with peaceful life (?), nothing else

In my opinion, ISRO's job falls into category 2, where you have good work, money, and life but no power or status. I have good work and money in the private sector (life is a hit or miss). So, I couldn't find any reason good enough to join ISRO. If I had the option to choose a job from category 1 (for example, all India services), I wouldn't have given a second thought to the idea. My current thinking is that if I have to live someplace other than my hometown, I would like to have either big money or power and social status.

It wasn't my original target

I attempted the ISRO exam to get some additional practice for the GATE exam. I had plans to get into some good colleges (IISc!) for further studies. ISRO's exam format coincides very much with the GATE's. So I thought that it would be an excellent opportunity to get some much-needed real exam experience. There was no other aspect to this fact. If I had taken the exam intending to join ISRO on selection, it would have been effortless to convince myself. But, in my scenario, I could not convince myself about the opportunity. Some may say that not everybody gets this opportunity, and I should have accepted it happily. I am sorry, but I don't subscribe to this viewpoint. I understand that the gods were kind enough to provide me with this opportunity. But I firmly believe that the same gods would also appreciate my reasoning about the decision to decline the offer.

For some people, these reasons may seem non-important. My personal circumstances and professional ambitions, and my self-confidence (perhaps over-confidence) contributed to my final decision to decline the ISRO offer. It is very much possible that I may regret this decision in the future when my life takes a downturn. I hope that this post will remind me of my present situation and bring me out of my grief. The decision taken is what matters now. I should focus on the next minute, hour, day, month, and years to come.

Wish me luck 🍀

Things I learned in Cisco

Sat, 22 Jan 2022 16:15:24 GMT

Note: This post was originally written on Jan 22, 2022 but I never posted it. Posting it now keeping the original authoring date, but with commit date of Aug 05, 2022.

After three years at Cisco, I finally decided to explore other opportunities in 2021. 22/01/2022 was my last day in Cisco. It has been an exhilarating journey. Even more so because it was my first such journey. I thoroughly enjoyed my time at Cisco, both professionally and personally.

Cisco provided me ample chances to work on a variety of problems. I started in a totally independent role exploring various open-source products and developing POCs integrating multiple open-source tools to the Cisco product line. I worked with OPA, ModSecurity, OpenRASP, etc. along with evaluation of Signal Sciences, K2 etc. We also worked on identifying differences between Snort's http module and ModSecurity, apart from developing POCs to identify supply chain attacks by evaluating the build artefacts generated in a CI pipeline.

Later I got a chance to work with a typical product development team when I switched to Snort team where I worked on the FTD integration for Snort, and helped in shipping high-quality code enhancing legacy Cisco firewalls. I also got an opportunity to contribute to some significant components to the very early stage Secure Firewall Cloud Native (SFCN) – Cisco's own cloud-based security solution – where I introduced an important component required for health monitoring in the SFCN clusters. Throughout my tenure at Cisco, I got opportunities to work on both the legacy (Lina ASA - a 20-year-old product) and the latest tools and technologies – Kubernetes, Docker, Terraform, Prometheus etc.

But, work was not the best thing about Cisco. The work was, of course, excellent, but what makes Cisco a dream company for many is its people. The culture that Cisco imbibes is pure gold. I don't know how the rest of the industry operates in this department, simply because of my inexperience. But for a novice like me, Cisco provided an ideal launchpad. I have many regrets about my time at Cisco, more on this later in the post. The people I met and worked with throughout my time in Cisco were exceptional in their work and, at the same time, were quite humble and down to earth in their dealings with me. I have also heard about some bad experiences from people working in Cisco, so I may have been fortunate in this aspect. Exceptions are everywhere, but they should not be the norm. And the toxicity is definitely not the norm in Cisco.

While I learned a lot of things in Cisco, there were a lot of opportunities that I didn't utilize while in Cisco. There used to be many hackathons, patentothons, cultural, literary, technical clubs, and sporting facilities in Cisco. Many of my friends were part of these activities, and some even led the organizing teams of these event. I never tried to contribute to any of these events. I had some personal and professional reasons behind this lack of enthusiasm. One of my colleagues criticized me privately for this on my last day - "Yash, I wish you had shown more involvement during the team building activities." I have been an introvert throughout my life, and changing one's personality is not easy. But this is something I will actively try to avoid repeating.

Now, you may ask, if everything was so rosy about Cisco, why did you leave, Yash?

Well, the reasons are many folded -

I am only 25. It is not a time to settle for me.
I was trying to break away from the traditional network security domain. I have been interested in distributed systems and cloud security. Still, my current team was not an ideal place for this. Switching to some other team inside Cisco was an option. But I decided to explore the industry from another perspective by changing the company.
Few more reasons that are not as important to mention here.

Anyway, Cisco will always have a special place in my heart for introducing me to the corporate world.

It is farewell for now. Au Revoir, Cisco!

2021 - A Bullish Year

Tue, 04 Jan 2022 16:55:24 GMT

2021 was a year of realigning priorities for me. I started the year with a set of goals, expectations, and an enthusiasm to achieve these, and here I am, ending the year with entirely different objectives and perspectives. There were a few ups and downs throughout the year, but collectively, these have transformed me into a more fitter, confident, enthusiastic, and curious version of my past self.

Health

If you read this blog regularly, you may know that I have been struggling to make health a priority in my life for the last few years. This year also started on the same note. I started with 65 kgs of weight, and by the end of July, my weight had increased to 71 kgs. I have been underweight for most of my life, but I probably became obese for the first time on the BMI scale this year. And I am not very proud of this, even though I always desired to gain some weight while growing up. I have long realized the importance of exercise in daily life but never acted on it seriously.

Nonetheless, this year I finally managed to work on it. Since September, I have reduced my weight from 71 kgs to 62 kgs. More importantly, I could bring my BMI down to an optimum level.

Another important aspect of health that I am working on is sleep. For some years now, I have been working on regulating my sleep timings and the time I spend sleeping at night. I have data about my sleep duration for the entire year. Using that, I produced the chart below, which shows that I have managed to sleep at least 8 hours on average (this also includes afternoon naps).

One of my last year's goals was -

Sleep at proper times - … I will try to sleep by 10 PM and sleep for at least 8 hours every day. I am aiming for an 80% success rate in this aspect.

I feel that I have managed to achieve this goal with a good level of accuracy.

Career

This year, I completed three years in Cisco. It has been a wonderful experience. However, I had a feeling that I should move on and start exploring the industry from a different perspective. So, I decided to go back to the interview table. In 2022, I will be joining a late-stage US-based cloud-security startup. I will write a separate post describing my experience preparing and attending the interviews.

I had to make a rather tough decision about my career this year. When I was preparing for GATE, I also decided to take the ISRO exam. The idea was to get some additional practice from the GATE perspective. After clearing the written test and the interviews, I got an all-India rank of 30 in the exam. As a result, I was offered a Group-A post of Scientist in NRSC Hyderabad. However, it didn't align with my vision and plans for the future, so I decided to decline the offer. I hope to write a long-overdue post about my experience with this sometime in 2022.

Blogging and Writing

The fact that I could not post this review on the first day of the year is a testament that I am struggling with writing. I am not in the proper mindset to write anything. I have tried to start small, but writing even a single sentence seems like a burden. Throughout 2021, I had a nagging feeling that I should write, but I couldn't find the enthusiasm to sit down and get going. Unfortunately, I don't even see myself writing anytime in the foreseeable future.

I notice that the procedure used to publish anything on this blog is very complex. Unlike WordPress, I have to battle with markdown, git, code editor, etc., to publish a post and make subsequent updates. The complexity associated with the static sites seems to have created a block in my mind. Unfortunately, developing a new website is no longer a priority for me. And I am not yet ready to accept the tradeoff between spending money on WordPress hosting and the comfort of writing in Wordpress's editor.

I planned to journal regularly in 2021. I started with that, but soon it turned into a logbook kind of activity. Instead of introspecting over my day, I started tracking each hour of the day, which was not the intention. So around July, I stopped doing it.

Reading

I had set a target to read five books this year. I completed that goal. I read a lot of PDFs this year, which can't be classified as books, so my reading statistics are very much skewed for the year. However, I didn't plan properly this year, so my efforts seem scattered and unfocused.

To be very honest, I was distracted throughout the year by the continuous flooding of information. My tendency to do everything with perfection further made things complicated. Another troubling issue was my resistance to making notes or summarising the books I read. I consider this an important activity because I find it challenging to recall otherwise.

I dabbled with Zettlekasten for some time this year. It is an exciting idea, but the time required to set it up is a big hurdle. It takes considerable effort to summarise a piece of knowledge, link it with other information, and then revise all of it. I may not be ready for it yet, but I will be trying it again in 2022 because I am very confident about the potential of this system.

I have reduced (or stopped) my Twitter usage to minimize information overload. I felt that passively reading others' views distorts my thinking and ability to generate original ideas. Although I am not really producing any exciting ideas, I want to keep my mind clean and free from distraction as much as possible.

Media Consumption

I logged my sleep data, work schedules, perceived productivity levels, perceived sleep efficiency, mobile usage patterns, etc., in a Google Sheet document. However, around mid-year, I started to get impatient with the whole manual setup and stopped logging my data. So, unfortunately, I don't have any way to analyze all these patterns now. I am totally disappointed with the lack of any standard tool that can collect at least mobile usage data while also ensuring privacy. Rescue time is something I have thought about, but I cannot convince myself to use that app due to privacy issues.

Below are my media consumption habits from 2021 -

For some reason, I watched a lot of content in the last two months of 2021. Possible reasons may be my acceptance of the new job offer or my truce with the weather. For whatever reason, I consider this year a controlled one in the binge-watching department.

What in 2022?

Build muscles - Now that my weight is at an appropriate level, it is time for me to build some muscles.
Read 24 books - I plan to read 24 books this year. I am yet to decide if I want to focus on a particular subject or diversify my reading.
Phone usage - Planning to keep it under 3 hours. However, I need to figure out how to export this data out of my phone.

As you might have noticed, most of my failures in 2021 have something to do with my habit of doing things perfectly. While it is good to perform any task as cleanly as possible, it sometimes becomes a big hurdle in starting up. In 2022, I hope to keep this tendency at bay as much as possible.

25, it is!!!

Sun, 06 Jun 2021 00:00:00 GMT

I turned 25 in the week gone by. I have been on the earth for a quarter of a century now, and the feelings are mixed. I love to write about myself, even at the risk of overexposing myself to the general world. This helps me in remembering my younger self. That, I think, is the best yardstick to judge myself against. Unfortunately, while I should write positive things, the tone of most of my personal experience articles tends to be negative. It is instinctive and something that I have not learned to control just yet. Nevertheless, I will consider myself successful when I look back at my life, and the memories make me smile. Some of my previous posts do make me smile.

The last one month or so has been tough on me. Suddenly, all my plans started to collapse, I started feeling depressed, and the feeling of being an imposter hit me hard once again. All this coincided with the second wave of COVID-19 in India, and it is easy for me to blame COVID. But there are certainly other reasons that have been building up for the last few years. The eruption of these accumulated worries from time to time drains me down. This has been the essence of my life for the previous few years since I started having an idea of self-ownership.

Borrowing from future and past

I see many of my friends and colleagues opting for higher studies abroad or in top institutes of India. While I also had such plans in the past, circumstances forced me to kneel down. I had hoped that I have adjusted to this reality. But a constant wave of social media posts from friends and people leaving the workplace has negatively affected my subconscious mind. Given that I decided with maturity and complete understanding of the situation, I don't know who or what to blame for my current mental state. I guess I just need to give it some time, and it will subside on its own.

I often tend to compare myself against others and feel low seeing people doing better than me. It is definitely not jealousy, but being competitive has so intrinsically become part of me that I can't help. Somewhere deep down, I also know that this comparison is futile. Different people can have different life situations, and life is not the same for any two persons. A better option for me is to compare against my younger self and see how much I have changed. That should be easy to do, given my habit of writing my experiences frequently. I don't know what is holding me back.

When I look back at my childhood, I see a weak, frail boy bullied by good and bad alike. Unfortunately, I never mustered the courage to fight them back. The bullying and my subsequent retreat into a cocoon have had a lasting impact on my life. I am still trying to reverse it. On the other hand, I still think that I am a born introvert. My life experiences might undoubtedly have given my personality its current shape, but it didn't all start with a blank slate.

The struggles of the present

One thing I am very proud of myself is that I never lost my curiosity. I am still inquisitive. I love to learn anything and everything that I can put my hands on (perhaps except artistic stuff). But what kills my enthusiasm is the fact that I don't utilize my learnings in any way. I tend to live my life in the same way I have been living it till now. There is no adventure whatsoever in my life, as I see it. It is as if whatever new I learn doesn't make any impact on my day-to-day life. When I think about it deeply, it appears to be a very unfortunate aspect of my quest for knowledge.

For instance, let us talk about reading. I agree that we should not read to remember, and that way, definitely reading has given me some new perspectives. I have read anywhere between 90-100 good quality books in the last one and half years. During a typical day, I get many insights related to whatever I have read. But when I sit down to articulate them in the form of a blog post or a personal note, I go blank. According to my introspection, it seems that most of my learnings are just superficial and I suffer from so-called illusion of knowledge. This reminds me of a quote by Daniel J. Boorstin -

The greatest enemy of knowledge is not ignorance, it is the illusion of knowledge.

Another thing is my short attention span. I can't focus on one thing for more than a couple of minutes now. This is one thing that has deteriorated over the years as I become more and more connected. I know that meditation will definitely help me with this issue. Still, I can't find the motivation to keep meditating continuously. I can start meditating any day. I can continue it for some days by using my willpower, but this is not sustainable. With a limited stock of willpower, I am bound to fail unless I make meditation a habit. I have almost cut myself from social media, instant messaging apps, and all other forms of communication to focus on the important stuff in my free time. Still, somehow my attention betrays me every single time.

For a couple of years now, I have been blaming my undecidedness to write regularly on so-called "writer's block." This is a complete sham. It has just served as a cover for my laziness and unwillingness. I often convince myself saying that I should first acquire knowledge. Then, one fine morning, I will wake up and write world-shaking articles. Of course, this is never going to happen, but why not, it is a good excuse to keep me happy with my comfort zone.

Conundrums of asking for help

There have been many people in my life who have offered to guide me and help me realize my goals. However, my reserved nature and the feeling of being an imposter have stopped me from opening up to anyone. I have been struggling with imposter syndrome for the last 3-4 years. I am not sure about the reasons yet. Still, I think my hesitation in requesting a "familiar" person for mentorship arises from the feeling that I might not perform as they would expect me to do. That would shatter my image in front of them. Therefore, my subconscious mind has decided that it is more optimal to keep the (false?) impression intact instead of getting the required help. That might also answer why I never share my articles with anyone. It is a fact that I don't feel very confident about myself and my work. It has affected me at the workplace, and perhaps it will continue affecting me in the coming years. The only way I can think of to get rid of this problem is to face it.

I have also often struggled to take criticism constructively. Somehow my egoistic nature and perfectionistic tendencies mixed with imposter syndrome make it challenging to learn from criticism. Just to clarify, I don't mean that I harbor any ill thoughts for the criticizer. It is something personal, and I don't have any words to describe it right now.

The illusion of competence

While reading has given me new and unique perspectives to think about the world, it has also given me an illusion of competence. I feel like I know a lot, but when it comes to utilizing the "earned" knowledge, I cannot interconnect my thoughts. I have seemingly different areas of interest, and that is reflected in my reading habits. My expectation is that diverse reading habits will result in a future me who will gather ideas from multiple fields and interconnect them to build a novel abstraction. I am not sure if this is a far-fetched desire and even possible for an average human. But I can't push myself away from this thought. Over the years, this has become one of many ambitions to get to that state. I don't know if I will ever reach that point, but I don't see any harm in keep trying. My only worry is how to stay away from the illusion of expert-level competence in a field, while I don't know actually have any expertise.

The political me

Those who know me in my personal life will attest that I am not able to stay loyal to any one side of the political spectrum. I have always see-sawed between left and right. Of course, this see-sawing has subjected me to "gentle" bashing from my friends adhering to either ideology. Famous Hindi poet Ramdhari Singh Dinkar very accurately points out the situation -

समर शेष है, नहीं पाप का भागी केवल व्याघ्र,
जो तटस्थ हैं, समय लिखेगा उनका भी अपराध।

But then, he says -

जो सत्य जान कर भी न सत्य कहता है,
या किसी लोभ के विवश मूक रहता है,
उस कुटिल राजतन्त्री कदर्य को धिक् है,
यह मूक सत्यहन्ता कम नहीं वधिक है।

Now, this is one of the paradoxes that troubles me quite frequently. While taking sides mean losing some of your individuality, not taking sides is also equivalent to not having a clear goal in mind, in a way.

Since when I started reading profusely, I have tried to keep this balance intact. I have made conscious efforts not to form strong opinions and not adhere to one particular ideology without looking at facts and data. The engineer's mind weighs everything in terms of data and facts and filters out personal opinions.

I read a lot about foreign policy, society, culture, caste, religion, economy, history, and the constantly evolving political structure of India and the World. But, unfortunately, most of the quality media is dominated by the left. I don't want to engage with keyboard-warriors on Twitter, WhatsApp, and elsewhere, so that leaves me with significantly less material to know the genuine viewpoint of the other side.

Over the years, I have realized that socially I tend to be on the center-left and economically, on center-right. But sharing political views in the public domain without proper research is not ideal. I want to consolidate my learnings and form well-informed and well-analyzed opinions on happenings in the world in general and India in particular before I participate in political discourse. It does not mean that I don't have any political opinions. Most events cannot be categorized as good or bad; it is just how one wants to perceive the world. Therefore, I constantly try to avoid getting influenced by what a random journalist with certain motives, sitting in a high-rise AC office, might want me to believe.

The CS guy!

Though I hated Computer Science a couple of years back, I have worked on my fears. Now, at least I don't have any understanding issues with the core CS subjects (except DBMS, perhaps. I don't like it; at all!) I didn't focus on understanding whatever CS theory was taught during my undergraduate (again, a lack of proper mentorship.) Although this doesn't harm me in my professional life, I want to get some kind of research experience at some point in my life. Unfortunately, it seems unlikely for me to enroll in a full-time graduate program in the foreseeable future. So, I am weighing my options to go for a Systems course via online studies. Assuming I get admission, one option is a sponsored graduate program (such as this one offered by IISc.) Another option is to go for an online MS degree provided by some US universities. I will surely miss the unique campus experience, but I hope it will still quench my intellectual thirst. I currently have some other stuff on my plate. I will try to finish them before deciding if and when I should opt for these options. I hope to make a call in the next two years in the worst-case scenario.

Originality

By originality, I mean a way to build upon something existing – some kind of improvement upon an already existing thing, idea, or abstraction. Almost everything humans have created has been an improvement over something preceding it, either man-made or natural. In my opinion, originality is a creative process to generate novel ideas improving upon existing processes, things, tools, and knowledge systems to make them more desirable, optimized, user-friendly, and insightful. Original need not be "original" in its literal sense.

While I have been working on my CS skills, it is not the field where I get my most original thoughts and ideas, despite being the subject where I spent 4 years of my life. I have met many amazing people in Cisco over the last 3 years. I am often awestruck by their ability and passion for continuously generating new ideas. Whether it is a Hackathon or a Patentothon, I have been fortunate to team up with people who can generate exciting ideas with surprising depth. However, I have thought about it and realized that I am not yet at that level where I can generate original ideas worthy of patenting or publication. Some people are just born with that kind of talent, and some acquire skills superior to any talent. With growing experience, I might get to that level someday, but currently, I am yet another average Indian software engineer. Read the last section of this article to know why I am content with this situation for now.

On the other hand, the fields where I tend to get many ideas nowadays are religion, politics, and history. One possible reason behind this might be that these subjects are my area of interest nowadays, and I read related books during my non-work day. So my mind gets a lot of time to consolidate this stuff.

It is such a contentious issue for me that I feel embarrassed talking about it. In fact, this is the first time I have actually given any thoughts on this topic. And I am glad that I did it. I wrote above my initial impressions about an issue I was unaware of until a few moments ago. I hope to get more clarity and insights into the real reasons behind my lack of enthusiasm to produce new ideas in the coming time.

Even if not entirely by choice, I am trying to embrace the slow-growth mindset, taking short-term hits for long-term gains. I hope this gamble will pay off, as I think it will be the right approach. Long-term gains far outweigh short-term benefits when it is a matter of whole life.

While I keep whining about my future and past and try to piece together my present, this quote sums up the way forward succinctly –

Thoughts on the Family Man 2

Sat, 05 Jun 2021 11:45:02 GMT

I binge-watched the second season of The Family Man (TFM) 2 last night. I was eagerly waiting for this season of TFM for a long time, especially after the highly gripping season 1. And I am not disappointed. While I watch digital content very selectively (young-adult fantasy depicting a dark and mid-age Europe and Russia and spy-genre Indian shows for occasional patriotic doses), very few shows have such excellent cinematography. It is the plot of TFM2 that made me ask some questions myself. While I don't have clear answers to them yet, I thought I will just list them down, along with some of my raw thoughts.

The plot is about the resistance of Eelam Tamils of Sri Lanka against the Sri Lankan government. The topic is riddled with complex matters of morality, humanity, human rights, ethnicity, language, majority-minority conflicts, and the right to self-governance. Sri Lankan conflict is familiar to most Indians because of India's involvement in the struggle in the early 1980s and 90s and the subsequent assassination of Rajiv Gandhi during an election campaign in Chennai. Many such issues are spread throughout the world – e.g., Israel-Palestine conflict, Rohingya issue in Myanmar, Nagaland in India. Specific questions that perplex me about these conflicts are -

Why do people fight for their ethnicity? How difficult is it to live alongside other people under a defined constitution prepared with a mutual agreement?
Why are modern nation-states reluctant to give autonomy or independence to such groups just to avoid bloodshed?
What are the causes of the origins of such conflicts?
Are there any examples where the final settlement was acceptable to both sides, and nobody left the table feeling betrayed?

I look for academic (Sociology, Anthropology, and Political Science, to be precise) answers to these questions. I understand that ground realities are often very different than what we learn from news media and books. Humans tend to have a natural connection to their perceived kinsmen. Why do you think nature has decided to keep a human baby dependent on her parents for almost 1/4th of her average lifetime? The bond we generate with our society, tribe, kin, and family stays with us forever, and we strive to keep them safe and secure at any cost.

I will give a controversial example - the Indian freedom struggle is a revered movement for most Indians, including myself. Still, for the then British rulers, it was a rebellion that had to be suppressed at any cost to save their empire. We despise the atrocities performed by the British on Indian people. How is that different from the horrors seen in the conflicts of the modern world (not just the people in power, rebels everywhere have also committed unspeakable atrocities). As I understand it, it is not a binary problem; there is no black and white in this. There are shades of grey and what amount of black you see depends on a lot of factors. Just be aware of the fact that the more powerful side need not be the necessary evil. Resolving any conflict requires some compromises from both sides. Until both sides are leaving the table feeling that they won, no negotiation will be successful. Remember -

We cannot negotiate with people who say what's mine is mine and what's yours is negotiable.

Looking back at 2020

Sun, 03 Jan 2021 10:19:58 GMT

While I thought of writing this review on 31st December, I couldn't overcome my laziness. It has been a persistent problem with me in 2020, which I haven't been able to fix as of now. On average, the year 2020 was a good year for me. It wasn't equally forgiving to me on every issue of my life, but it was most undoubtedly lenient on the issues I consider most important at this phase of my life. As for most people, it was a year of self-retrospection for me as well. I identified some new goals in my life, forgot some others, compromised with some failures, and learned the value of loved ones.

Like previous years, I have identified some critical areas of my life, where I am putting more focus year after year. It is easier to focus on one topic at a time. I have decided to omit some parts of my life from this write-up because I am not yet ready to reveal them to a specific set of people (although very few people read this blog now, thanks to my irregularity, I prefer to stay risk-averse).

Health

2020 belonged to the pandemic. So it is most prudent to start with the topic of health. For the last some years, I have been struggling with a lot of health-related issues. Although this year was a much better one for my body and mind, I am still petrified of my health issues. It seems that I am a worry-magnet, and the worry of catching coronavirus was the biggest of them all this year. I spent almost 3-4 months in this worry. It took some toll on my health, but in the end, I was able to come out of it, thanks to family.

I should give part of the blame to my laziness. I had stopped all my learning activities in the mid-to-late year (perhaps I was exhausted). And that gave way to evil thoughts in my mind. As they say, "An idle mind is the devil's workshop!". Thankfully I got hold of the situation while it was still manageable.

This graph shows how my sleeping times changed throughout the year. I lost track in the third quarter of the year. Anyway, in December, I seem to have come back to the correct path.

This chart doesn't convey certain things very clearly. First, it shows the average amount of time I slept in a particular month. However, that time includes the afternoon nap also. And in the mid-year months, where you see 9+ hours of sleep, it means that I was sleeping quite a lot in the afternoon and less at night. In the last three months of the year, I have been trying to regulate my sleep timings at night, and I am quite satisfied with my progress. I have observed over the years with my sleep experiments that whenever I start sleeping less at night, I tend to exhaust myself in the next couple of months. This theory got validated this year when I exhausted myself from June onwards, and my health issues reappeared after sleeping significantly less in April and May (night sleep).

I actually did some exercise this year. I bought a gym subscription this year, but unfortunately, that became my worst financial decision in the last several years because I left Bengaluru before the government imposed the lockdowns. However, I did some light exercise at home. That didn't do wonders, but at least I was able to keep my BMI constant throughout the year. Even after eating delicacies prepared by Mom all year!!! 😍 Hopefully, I will be able to start running in 2021, even if only on weekends.

In November, I suffered from severe back pain. After getting an ergonomic chair, things seem to be in control. But that incident reminded me of the importance of exercise. I will try to include some simple back exercises in 2021.

Family

The year 2020 was quite an unusual year on this front. Since I started at NITC in July 2014, I have been away from my family for most years. But thanks to lockdown, I got to spend time with my family this year. I had anticipated lockdowns in India quite early, so I left Bengaluru in March just before the government imposed lockdowns in a "shock and awe" manner. So I reached home on time, and that was the best decision I have taken this year. If not for my family, I don't think I would have been able to handle pandemic-induced health issues.

I have been away from my brother for almost seven years now. More importantly, these seven years were the life-defining years for both of us. It turns out that he and I think very differently. I guess that our thinking results from the different environments where we spent our early adulthood. It was initially very tough to agree on something, but we slowly learned to agree to disagree. Staying away from family has made me more careful and risk-averse in some sense. I tend to (over) analyze everything, and that causes delays in decision making. On the other hand, he is quite free-minded and rebellious. How different we both have turned out to be!!!

I don't mind this variety, though.

Reading

I had set a target to read 20 books this year. In the end, I managed to read a lot more than 20. Mid-year, Around July/August, I started losing track of the books I was reading. As I mentioned in my last year's review, I was getting disillusioned with Goodreads for some time. This year I tried to use it, but ultimately I didn't log the majority of my reading anywhere. I don't know if this is a good thing to do. I will try to keep it this way for some more time and then analyze the data-collection pros and cons.

I have identified some issues with my reading pattern. I tend to read the books passively. That makes it quite challenging to retain knowledge for the long term. On top of that, I don't make notes as well. So, I cannot even revise the content quickly without going through the whole book again, which is counter-productive. It is a childhood habit, and I assume it will be tough to break this habit. However, whenever possible, I now actively remind myself to question the book and not take everything written in the book for granted. It will take some time (perhaps years), but I am confident about changing this tendency.

I read a variety of books this year. Most of these books were related to humanities subjects like political science, history, etc. I got interested in tech quite late in my life. Before that, humanities was my favorite pastime. Once I entered the tech field, I somehow couldn't continue this. But this year, I found myself distancing from technical topics somewhere around June. There is a reason behind that disinterest that I would like to keep private for now. At the same time, I struggled with my health issues, so I decided to start with humanities again. Looking back at that decision, I feel it was a superb choice. I have learned so much about India, its history, and the present. Somehow, I was able to ask questions about why a particular thing was done the way it was done. I was never able to think like that while reading a technical book. Does that make my knowledge about tech hollow? I don't know yet. But I will be actively looking for an answer in the coming times.

One subject, where I expected to glide through was Economics. I mentioned in my 2018 review that I find Economics interesting and want to learn more about it. I couldn't do that in 2019, but this year I decided to pursue that goal. So I started with the most fundamental books - NCERT class 12th Microeconomics and Macroeconomics. The result is that I have lost my interest in learning economics now (hopefully temporarily). Either the books are written horribly, or I didn't put enough effort. It took me ages to complete these two books, and still, I don't understand a lot of the concepts. I think it was the wrong strategy to start with these books. I should have watched video lectures by some University professors. Perhaps I will do that this year, whenever I get time from my other goals.

Another subject, which caught my interest was Anthropology. I had read the excellent "Sapiens" book by Yuval Noah Harari a couple of years back, and that was a thought-provoking read for me. So I decided to read further on the topic to answer some of the questions that I had about human society and why certain cultural elements got evolved in the way they are at present. It is quite an interesting subject, specifically social and cultural anthropology.

Below are some of the superficial statistics about the books that I read this year -

	Year Goal	Achieved
Read/to-read	20	69
Numpages	7000	16279
pages/day	19.13	44.48
days/book	18.30	5.30
book/month	1.67	5.75

Once again, these numbers are not accurate indicators of the quality of reading. Not all of the books were worth reading. I consider around 25 of these books as a waste of my time. Some books were as small as 100 pages, while some were so good that I wanted more even after reading 800+ pages.

Now, the question is how many of these books benefited me. I plan to reread some of these books and prepare some notes after/along with the second reading. I have tried writing on paper, but my perfectionistic tendencies stop me from "spoiling" the paper. So I will make the notes on Onenote this time and write them down on paper later. Rewriting on paper is a compulsion because my mind seems to be having issues recalling digitally-read content. It is one reason why my Kindle is inside my drawer for a very long time.

One of my wishes for 2021 is to read The Story of Civilization.

Blogging and Writing

It is quite evident from this blog's activity in 2020 that I didn't write very actively. I am going through a challenging phase (imposter syndrome?) in my tech journey that is difficult to describe. I started this blog when I was actively learning new things in tech. But now, I have almost hit a plateau. If I phrase it in another way, I am not learning enough technical content now. I don't think it is related to my work, but somehow, my attention has been on other things lately.

I have not been able to write anything on paper as well. I did some analysis on this issue, and it turns out that I have perfectionist tendencies, as I mentioned in the last section. I try to do everything with perfection, and that doesn't work most of the time. You can't always have everything perfect in real life, although aiming for perfectionism is not a bad thing. But when this goal becomes the process, it starts creating problems. And I am facing the same issue. Now, whenever I sit to write anything, I remind myself at regular intervals that I don't have to be perfect all the time. It seems to work till now.

Anyway, in 2021, I will write a lot on paper and perhaps on this blog. I am still contemplating if I should write long-form essays on some contemporary issues on this blog. Maybe I will create a new WordPress blog for that kind of stuff and keep this blog for my technical content and everyday life-related posts.

I figured out one more issue with my thought process. Given a topic, if you ask me to write a summary of it, I wouldn't be able to articulate my thoughts. Something that Richard Feynman was very good at. In other words, I struggle in explaining my knowledge to others. I guess this means that I don't understand the topic well enough to write anything about it. It might also mean that I have some issues in thinking clearly. I hope, with practice, more active reading, and regular writing practice, I will win over this issue.

Work

It isn't easy to summarise my work experience. It's my third year at Cisco now. The team in which I was working for the last two years got dissolved. However, higher management was quite helpful in transitioning to a new team working on Snort development. So, I will be getting a much-needed product development experience now. Till now, all is good; however, product development is slightly weird. I need to give it more time before drawing any conclusions. Since I have joined the new team, I feel that my day-to-day learning has suffered during work hours. I hope it is a temporary phenomenon and I would be able to overcome it soon.

The COVID outbreak has spoiled some of my plans, and I am unsure if my familial circumstances would allow me to opt for those plans in the near future. I reoriented my goals to adjust to the changing conditions. But whatever happens, I will try to utilize the available resources to the maximum extent possible. Hopefully, everything will settle in the right place soon.

Media Consumption

I seem to have been able to conquer the addiction to social media. I had been trying to do this for 3-4 years, and this year, I feel that I was in control for most of the time. Whenever I lost myself in Twitter/Quora/Reddit, I found it easy to cut-off myself from these websites without any withdrawal symptoms, which is a very pleasing experience.

Facebook deserves a special mention here. I have been entirely out of touch with Facebook for the last couple of years, partly due to self-control and partly due to the bad press Facebook has recently gotten. One thing that has helped me is that I don't have any content on my newsfeed because I took some time to unfollow everyone. As there is no content to scroll, I don't feel the urge to stay longer on the website. I log in occasionally to the website just for checking the notification and friend requests.

Coming to the media content consumed by me, these are the movies that I watched this year. I did a lousy job logging this stuff, so this data is not comprehensive. Of course, I watched a lot more movies than this.

I watched many web series during the July-October period. I can recall a few right now - Sacred Games 1 and 2, She, Flesh, Bard of Blood, Mirzapur 1 and 2, The Witcher, Sex Education, and Paatal Lok. In short, I wasted a lot of time watching useless content. Sensory satisfaction, peer pressure, and the urge to run away from my boredom are to blame for this binge-watching time.

I am using Letterboxd to log the movies, but there does not seem to be a free service that offers both movies and web series logging. I wish somebody will make one in 2021.

I use an Android phone and use Digital Wellbeing to monitor my phone usage. Unfortunately, I can't export my usage data from the Digital Wellbeing app. This data gets lost because I switch ROMs frequently. However, according to my liberal estimate, I was able to keep my per day phone usage under 4 hours on most of the days. There were some outliers, but mostly, my phone addiction was relatively under control. However, I would have liked to keep it around 3 hours per day.

What in 2021?

This section is a new addition to this year's review. I usually don't set explicit targets and certainly don't share them in public. So this is something different. I am sharing my goals here. I will take this as a benchmark when I sit to write my year review for 2021. I am hoping for at least a 60% success rate.

Publish six long-form essays - The plan is to learn the skill of articulating thoughts. I don't have any particular topic in mind. Perhaps a mix of History, Political Science and Tech. It can be in the form of essays or an explanatory post, or anything else. Ideally, every two months.
Read five fiction books - To learn how other people write fantasy. I hope to be able to read it for learning, not just for fun. You can follow my progress here.
Sleep at proper times - Now that majority of my work meetings are expected to get scheduled during IST, I will try to sleep by 10 PM and sleep for at least 8 hours every day. I am aiming for an 80% success rate in this aspect.
Phone Usage - Keep it under 3.5 hours per day (Aiming for 80% success)
Write a Journal - I have tried it earlier also but failed. This year, I will try to document only important events in my life. At least once a month should be good enough. The more the merrier!
Regular exercise - I need to reduce my BMI to 22 from 23.3. That requires me to shed at least 4 kgs of weight.
- Weekend running (Aiming to do it on 50/104 weekends, accounting for winters)
- Mild 15-minute exercises at home two days a week (Aiming for 80% success)
- Meditation on the two days of the week (Aiming for 90% success)
██████ (redacted) - I want to achieve this one badly.

Note: A number of these goals depend on how many more months I get to spend at home, after June.

2020 was quite a year. I hope humanity has learned its lessons. While industrialization and globalization have brought us a couple of hours away from each other, it has become equally easy for pathogens and germs to travel and reach farther ends of the world. I like to equate this situation to the disease-carrying Europeans who went to the Americas in the 17th century. Nations could practice some carefulness and proactivity in dealing with such problems. It is not the last pandemic to ravage the world, for sure.

I expect to achieve some of my goals in 2021, but I won't deny a potential role of luck or chance (whatever you like to call it). Hopefully, the gods of chance will be on my side.

Two Years at Cisco

Wed, 16 Sep 2020 16:33:46 GMT

I completed two years at Cisco in August recently. I think it is a good time to jot down my learnings and experiences from the past two years. It was my first corporate job; however, I did not land up in a typical software development job. Hence, my experiences are quite different and unique in a sense.

I joined Cisco in 2018. As it usually happens with university recruits, I was also randomly assigned to a team in the Office of CTO in Security Business Group (SBG) in Cisco. I got assigned to a non-product team that worked in a semi-research environment to develop PoCs demonstrating the integration of future-looking technologies with the existing Cisco portfolio. The nature of this work gave me some exciting skills and exposure to latest trends in industry, but it came with its own set of disadvantages. I will discuss some of them at the end of this article.

Take Ownership of your work

Every member worked on a reasonably independent project in my team, directly supervised by a Principal Engineer or a Technical Lead. Most of my superiors (including my manager) were located in the USA, so there were many early-night meetings involved. This also meant that nobody watched (micromanaged 🙊) me, and I was given quite a bit of autonomy. I really appreciate this aspect.

I learned how to come out of a problem when there is nobody to help or guide you. Of course, I asked for help whenever needed; the freedom I enjoyed at work was unparalleled.

During two years, there were many occasions when some of my design and development choices were questioned. I successfully defended some of them and had to change the others. In the process, I learned the caveats in my understanding and the importance of others' perspectives while solving difficult problems.

Developers should have experience with operations

As I mentioned above, most of my projects were individual projects. Naturally, the responsibility of setting up and maintenance of the build-environment also fell on my shoulders. My experience with Linux system administration from college time came to my rescue here. In the process, I learned how invaluable the skills of DevOps can be for a developer.

There is a lot that I don't know yet, but my basics have grown strong enough to grasp any new technology in a reasonable amount of time over the past two years.

Communications

I have never been a good speaker, and talking to strangers has been one of my weakest skills. After joining Cisco, this aspect of my personality has changed considerably. Attending regular meetings and speaking about my ideas and work gave me confidence. There are still some caveats with my public speaking skills. I hope that I would be able to overcome these challenges with more experience.

Innovation

The industry provides you access to the best minds in your field. You get to meet wise people working and using their knowledge and expertise to generate ideas that can change the world (or at least the industry). It motivates you to compete with them. Although I have been unable to generate any patentable idea in the past two years, I have been invited to participate in the discussions, which led to patent applications. My biggest takeaway from these discussions was that breadth is more useful than depth when it comes to idea generation. Knowing various fields and connecting two seemingly different domains is a skill whose importance cannot be overemphasized.

Another thing I learned is that aiming for perfection doesn't help anyone. To keep waiting for that best idea to pop up in your mind is illusionary and will never happen. I have been a victim of this tendency to perfectionism the whole of my life. For the past couple of months, I have been actively trying to avoid this habit. My inconsistency in regularly writing on this blog can also be attributed to my search for the ideal blog post.

In my opinion, quantity is better than quality. The same is true with the generation of ideas. If you keep waiting that you will get the ultimate idea one day, believe me, it will never happen.

Take a step back & rethink

It happened quite a few times that I started looking for a solution to a problem in one direction. After spending a considerable amount of time, I figured out that the direction was wrong or worse, my understanding of the problem was wrong. In such situations, it is better to backtrack and think again. It also helps to ask someone for help. Sometimes, others can give you a different perspective that you might not be able to see.

Don't hesitate to brag about yourself

When I joined Cisco, my colleagues suggested me to set up 1-1 with my manager to stay in touch with him and showcase my work. I hesitated. I thought that what is the point of talking about my work when it is visible to everybody through weekly meetings. Now I understand, it is such a crucial step which shouldn't be ignored at all. Keep a brag document and use it to keep records of your work, achievements, and accomplishments. I never needed it till now, but I plan to use it in the future.

Coming to downsides, I have only one qualm. As I mentioned above, my team was not a product team. So I missed out on some of the necessary product development experience. I believe that having some experience developing and shipping production-ready code is an absolute requirement to move towards my career goals. Thankfully, I have switched my team now, and now I will be working on a full-fledged security offering of Cisco.

So, these are some of my insights from the past two years at Cisco. I want to share a lot of other thoughts, but their time hasn't come yet. I will definitely share more of these as I spend more time in the industry and tech.

Cassandra - A Decentralized Structured Storage System

Fri, 08 May 2020 01:51:55 GMT

Cassandra is a distributed storage system that can spread over thousands of nodes and store terabytes of structured data. Cassandra was developed at Facebook to solve performance issues during searches in Facebook Inbox. Cassandra can provide a highly available service without a single point of failure.

Cassandra borrows some of its architecture choices from Google's BigTable and Amazon's Dynamo. In some ways, Cassandra resembles the traditional databases, yet it does not support the relational data model completely.

Data Model

Cassandra uses a tabular data structure like relational databases. A table in Cassandra is a distributed multi-dimensional map indexed by a key. Every row operation is atomic per replica, no matter how many columns are being read or written into. Cassandra groups columns into sets called column family. A column family is of two types - Simple and Super (column family within column family). Cassandra can sort columns using name or time. Time-based sorting is useful when used in a setting like Inbox searches where most recent messages should be displayed first.

Cassandra provides following methods -

$insert(table, key, rowMutation)$
$get(table, key, tableName)$
$delete(table, key, tableName)$

Architecture

A data storage system should satisfy some requirements - data persistence, scalability, membership, failure detection and handling, data partitioning, request routing, among others. Cassandra uses a variety of techniques to solve these issues.

Routing

Any node can attend an incoming read/write request with a key. Each node in the Cassandra cluster knows about other nodes. The serving node uses this information and the request's key to determine the route of the request to the appropriate node.

In the case of write requests, the system routes the request to all replicas and waits for a quorum of replicas to acknowledge the completion of the writes.

For reads, the system either routes the requests to the nearest replica with required data or forwards the request to all the replicas and waits for a quorum of responses before replying. Which method, the system utilizes, depends on the consistency requirements of the client.

Partitioning

Cassandra uses consistent hashing with an order-preserving hash. The consistent hash function is used to hash the data key to generate identifiers. The output range of the hash function is treated as a ring (the largest hash value generated wraps around the smallest hash value). Each node is assigned a random hash, which becomes its position on the ring. Each node is responsible for the region in the ring between it and its predecessor node.

For each data item with a key, its hash is generated using the key. The ring is traversed clockwise, and the first node with the position hash value greater than the item's hash value is assigned to the data item. This node is deemed the coordinator of the key.

However, consistent hashing can result in an imbalance in the load distribution and non-uniformity of data distribution. Cassandra solves these issues by periodically analyzing the load information in the ring and repositioning the lightly loaded nodes on the ring to alleviate high load nodes.

Replication

Each data item is replicated at $N$ nodes in the ring. As mentioned above, each data item's key is assigned to a coordinator node, which is responsible for the replication of data items falling within its range (the region between it and its predecessor node). For fault tolerance, in addition to locally storing keys, the coordinator node replicates these keys at the $(N-1)$ replicas on the ring. Replication policies such as Rack aware, Rack unaware, Datacenter aware are used for this purpose.

Leader Election

Cassandra uses Zookeeper for leader election and fault tolerance. Whenever a new node joins the cluster, it contacts the leader who tells them what ranges the node is responsible for. The metadata about ranges of a node is cached locally as well as on the Zookeeper instance.

Membership

Cassandra uses Scuttlebutt - an anti-entropy Gossip based protocol to disseminate the membership information inside the ring. Whenever a new node joins the system, it calculates a token for itself. This token is gossiped around the cluster. That's how each node in the system knows about the membership and positions of other nodes in the system.

Failure Detection

Failure detection is a mechanism by which a node can locally determine if any other node in the system is up or down. Cassandra uses a modified version of $\phi$ -Accrual Failure Detector. The basic idea is that the failure detection module emits the suspicion level of a node instead of a binary up/down status. This suspicion level is $\phi$ . The idea is to represent $\phi$ on a dynamically adjustable scale, which reflects network and load conditions at the monitored nodes.

Every node in the system maintains a sliding window of inter-arrival times of gossip messages from other nodes in the cluster. The distribution of these inter-arrival times is determined, and $\phi$ is calculated. Cassandra uses exponential distribution as an approximation for determining $\phi$ . The original $\phi$ -Accrual Failure Detection Algorithm used Gaussian distribution.

Scaling

Whenever a new node joins the system, its token is generated such that it falls within the range of an existing heavily loaded node. This results in the new node splitting the range of the old node. The old node transfers some of its data to the new node using kernel-kernel copying techniques.

Local Persistence

Cassandra uses a commit log as well as an in-memory data structure to store the data. Each write is first committed to the commit log. Only after successful write into the commit log, the data is saved in the in-memory data structure. When the in-memory data structure crosses a predefined threshold, it is dumped to the disk along with an index file for fast lookups. A merge process runs periodically to merge these disk files.

A read operation first queries the in-memory store. If data is not found there, then a disk lookup is required. To avoid looking into multiple files, a bloom filter, summarizing the keys in the file, is also used. The bloom filter can also be used to check the key existence.

Paper Link:- Cassandra - A Decentralized Structured Storage System

Epidemic/Gossip Protocols

Tue, 14 Apr 2020 03:08:15 GMT

Last week, while reading the book Designing data-intensive applications, I came across the term "Gossip Protocols." The title was quite intriguing; hence I search for it on Google. It turns out that it is a communication protocol. It is sometimes also called the "Epidemic Protocol."

We are facing an ongoing pandemic called COVID-19. The term "Epidemic Protocol" caught my attention, and I started wondering how the knowledge of epidemics is going to be useful in computer systems. It turns out; these protocols try to emulate the spread of a virus to effectively communicate the information to all nodes in a distributed network.

A virus spread quickly and robustly. Our goal in a distributed system is to spread the information/updates as quickly as possible without burdening the network. The epidemic protocols try to bring these ideas from epidemiology to distributed systems.

I use both terms (Gossip and Epidemic) interchangeably in this post.

Analogy to a real epidemic

Let' take a close look at how a virus spreads. I'll explain it using a small sample of five people. We assume that, initially, none of these people is infected. Now, because of some external factors, one of these (say $A$ ) got infected with the virus. We say that $A$ is infected, and the remaining four people are susceptible to infection. $A$ followed the advice of doctors and isolated itself from the group. Now we say that $A$ is removed (either because he has the infection, but is not spreading it, or because he is recovered).

Now, let's extend this analogy to a network. In a network, we have multiple nodes. These nodes are classified using the terms – infected, susceptible, and removed. The infected nodes try to spread some information by periodically selecting some peer nodes from the network. If a node is susceptible, that is, it does not know the said information, then after the selection and transmission of information by an infected node, the susceptible node also gets infected and starts spreading the information. A node is said to be removed, if it already knows the said information, but is not spreading it because, for example, all its peers already know the information, so there is no need to keep spreading it – the so-called herd immunity).

Some theory

The concept of the Gossip Protocol is not something new. The 1987 paper Epidemic algorithms for replicated database maintenance is considered seminal on this topic. The Gossip Protocols were initially used to maintain consistency in replicated databases for efficient data communication. Later, these protocols found their usage in other areas such as service discovery in a distributed environment and maintaining node memberships as well.

Usually, these protocols work as follows -

A node ( $A$ ) in the network randomly selects another node with which it wants to share information. Here, the assumption is that each node in the network either maintains a list of all the other nodes or gets the information from a centralized server.
On receipt of information, the receiving node ( $B$ ) processes the information.
In the next round of this process, both $A$ and $B$ again select nodes randomly and transmit the information.
These steps repeat periodically until the information is disseminated to every node in the network.

Methods

In the paper mentioned above, two schemes of epidemic protocols were analyzed -

Anti-entropy

In this scheme, a node $p$ randomly contacts a random partner $q$ from the current population. The nodes $p$ and $q$ engage in information exchange to resolve any differences between them. The updates known to $p$ but not known to $q$ are transferred using different strategies (push, pull, and push-pull).

As it turns out, anti-entropy requires significant network bandwidth, because it needs to send the complete database contents to other nodes for resolving any differences. There are many approaches, such as sharing checksums, Merkel trees, maintaining a recent update list, etc. that can be used to reduce the bandwidth requirements in the anti-entropy algorithm. These strategies allow the sending node to know what updates the receiving nodes require to become consistent.

It can be proven that this algorithm guarantees the eventual dissemination of information. The number of updates sent in this scheme is not bounded, so there is no termination.

This scheme is equivalent to the SI model (simple epidemic) from epidemiology. The term SI stands for susceptible-infected (same as explained above). A node is always susceptible or infected.

Rumor mongering

As the name suggests, this scheme works similarly to how rumors spread. Initially, all nodes are ignorant of a rumor. When a node learns about some updates, it becomes a "hot rumor." While a node holds a hot rumor, it periodically chooses another node at random and pushes the rumor to the other site. When a node has tried to share a hot rumor with too many nodes that have already seen it, the node stops treating the rumor as hot and retains the update without propagating it further. Rumor-mongering requires very less network bandwidth because it needs to send only recent updates to other nodes.

The equivalent of this in epidemiology is the SIR model (complex epidemic), which stands for susceptible-infected-removed. A node can be susceptible or infected or removed.

Because of the removal of nodes, the number of messages transmitted in this algorithm is bounded. However, because of this, there is a slight chance that some updates might not reach all nodes. So it does not guarantee eventual consistency.

There are two strategies to decide when a node should be removed -

The analysis of gossip algorithms focuses on designing strategies on how to select the best peer group to share the information with.

If you want to get started with this topic, here I recommend some papers that are quite fundamental when it comes to an understanding of how gossiping works:

Epidemic Algorithms For Replicated Database Maintenance
Gossiping in Distributed Systems
Randomized Rumor Spreading
The Promise, and Limitations, of Gossip Protocols
Gossip-based Protocols for Large-scale Distributed Systems - Read the first chapter of this book to get a basic understanding of Gossip protocols)
A gossip protocol simulator

P.S. - This is my first attempt to read and summarize CS research papers. I have intentionally covered only a small part (first few pages) of the paper (first reference in the above list) here, as I am still figuring out the best way to read and summarise. I am confident that with time and practice, I will get better.

If you find any scope of improvement in current content, please let me know through email or comment box below.

2019 - Year in Review

Tue, 31 Mar 2020 06:09:47 GMT

It has been three months since 2020 started. I was not able to keep up with my yearly habit of posting year reviews due to some reasons. One primary reason behind my negligence is my disinterest in writing on this blog. I felt that it was because the old layout of this blog had been too recurrent for me. So, I decided to fix that issue before thinking about writing.

The year 2019 was not a great year for me on many fronts. It was a year that reminded me of my limitations and brought me out of the bubble, that self-help books created for me. In retrospection, I find that I started a lot of things, with the intention of learning, but couldn't achieve any of the targets.

Like, I structure these posts every year, I have divided this post too in many sections. It makes it easier for me to focus on various aspects of my life in the past year. I will start with the best parts of my life.

Reading

I had decided to read 26 books in the year. I finished the year with 28 books. Out of 12 months, I did not read any book in January, November, and December. I finished two of the fiction series that I wanted to read for years - Harry Potter, and Percy Jackson. These two books are the reasons that I was able to finish 28 books in 9 months. It took me very little time to finish these book series. Other books (mostly non-fiction) took me ages to complete.

I am not sure if I am getting any value out of non-fiction books right now. I do not take notes and forget most of the lessons in a few days. Not sure if this is the right strategy while reading such books. Hence I have decided that I will reduce the number of non-fiction books for some time and put more attention to technical publications and CS books.

Though I am confident that my note-taking skills will not help in reading technical books either, I do hope to improve over time.

One of my reading goals from last year was to read a research paper every 15 days. I succeeded in the first quarter of the year and failed miserably after that. I started with a good intention and kept reading for some months. Then self-doubts began popping out. What value am I getting out of this?, was the question in my mind. I started this exercise because I have specific goals in my mind, and I wanted to improve myself academically. But when you are not able to answer such a question even to yourself, then something is wrong.

This failure has taught me some crucial lessons. I tend to start things without thinking about the end goal. It is the right approach when learnings from the process is more important than the result. But I was not able to focus on either of them. I lost my focus when I realized this.

I have come to understand the value of proper note-taking. I have read so many good books, but I do not remember any lessons from these books, because -

I read passively and try to finish the book as quickly as possible.
I did not try to implement the lessons in my day to day life while reading.

I lost my interest in reading news and blog articles also. I lost my interest from social media and, in general, the Internet entirely. I preferred to sleep as much as possible. That has its side-effects. Read on.

Blogging and Writing

I had decided to write regularly in 2019. I didn't. It seems that failing was a norm for me in 2019. This section was my worst hit interest in 2019. Somehow, I was not able to gather the concentration required for writing. I used to get sudden motivations for a few days, I wrote during those periods. The rest of the time, I just thought that I should write. I think I suffered from the so-called Writer's Block.

Health

My analysis says that all my failures last year can be traced to this section. I was sick for almost a quarter of the year. Repeatedly getting ill and the time required for recovery took a toll on my body and mind. I lost all my focus and will power to do anything.

The exercise was again an issue. I certainly put more effort than last year, but that effort was scattered throughout the year, for small periods. So, I do not notice much difference. However, I continued to walk whenever I got time.

Work

It was not a very exciting year on the work front. I continued to work in Cisco this year. The work was neither very interesting nor very dull. The nature of my job is such that the exciting work shows up from time to time. With no fixed deadlines and no fixed way of doing things, it can be quite satisfying and quite frustrating at times.

I lost my interest in social media altogether. I opened Facebook only to see the notification once in a while, sometimes after months. I got disillusioned by Twitter this year, because of the toxic political content there. I was never a regular user of YouTube anyway, so that was never a concern. However, I went to great lengths to delete every Google app from my phone, especially Gmail and YouTube. The email had become a new Facebook for me. I was checking emails every 15 minutes. So I signed out of Google account in my primary browser and restricted my email activities to a secondary browser. It helped me in keeping the urge to check email in control.

But this sudden disinterest has created a void in my typical day, which is quite reasonable given how much my life depends on the Internet. I am yet to figure out how to fill this void. Reading books did not help, however sleeping and watching movies helped for sure (however, not without its own set of problems).

Travel

I was somewhat dissatisfied with myself for not traveling enough. This year, I had decided to travel extensively. But my health betrayed me. Out of 365 days, I was able to take the time of 4 days for travel. I went to Coorg and Chikmagalur with friends. Both trips happened during the wrong weather conditions. But the circumstances were such that I could not deny to my friends.

Miscellaneous

My daily routine, more or less, remained the same. Though I couldn't wake up early in the morning most of the time, the reality is that I was not trying. My only intention behind waking up early in the morning was to meditate. I did that, however sporadically.

One other goal in 2019 was to get rid of procrastination. I did not care about it at all. Somehow, I was too busy fixing other issues in my life, that this seemed like a minor problem. Anyway, I did not attempt to do anything on a personal level, where procrastination would affect me. On the work front, at one point or another, you have to work and show results. So there it was never apparent. I tried many things to reduce the impact and influence of technology in my life. Although I consider myself successful to an extent, there is still scope for improvement.

In 2018, I was in full praise for to-do lists and the concept of maintaining goals. 2019 has busted all those praises. This year, these techniques did not work for me at all. Anyway, 2019 was one of those times when nothing worked for me, so this failure does not come as a surprise to me. Though, I still believe that this is something that can genuinely help me with some of my issues.

If I had written this post at the beginning of the year 2020, I would have written some wish for the year 2020 at this place, as I always do, but this time is so gloomy that I am not able to think of anything good. I just pray to God that everything gets all right soon.

Being with family during this time, has given me some mental stability. But, it is also true that facing such circumstances and coming out triumphant, is what shows the human character.

Automatic HTTPS Certs Using GoDaddy and Gitlab APIs

Tue, 23 Jul 2019 11:10:28 GMT

Before I start, here is some technical information about my website -

Hosted on Gitlab and usage its CD for automatic deployment, which is triggered on a git commit.
The domain name registrar is GoDaddy.
Let's Encrypt as the TLS certificate provider for my domain name.

Let's Encrypt provides certificates for 90 days only, so I was forced to set up the certbot every three months, then generate the certificate and manually deploy the certificates to GitLab. It was frustrating because in three months, my shell probably will not remember the command I used last time, and I was too lazy to document the process somewhere.

Then I came across a project, which was trying to do a similar thing with other domain name registrar. I thought that this is an excellent setup to adapt and modify it to suit my needs. So I decided to write my module for automatic deployment of Let's Encrypt certificates to GitLab.

How does Let's Encrypt work?

Before Let's Encrypt can generate the certificate for the domain, it requires the user to prove domain ownership. Let's Encrypt provides two methods to do this task -

Updating the DNS records of the domain registrar (DNS-01 challenge)
Adding an HTTP resource under a well-known URI on the website (HTTP-01 challenge)

Using the second method requires me to add a file on my website, and I did not want to do that. Moreover, if in the future, I decide to host some other service, which does not have a website, then this method will fail.

The first method requires that the domain configuration on the domain registrar is modified. GoDaddy provides a robust API, and it is not difficult to utilize that API to automate specific tasks. The probability of changing the registrar is quite low, at least for the next few years. It will be easier to generate the certificates for any subdomains, as all the subdomains will also be hosted under the same domain registrar.

How does the tool work?

So here are the steps:

Call certbot with all the domain names
Invoke GoDaddy API to update the DNS records as indicated by Certbot
Wait for 10 minutes for DNS changes to propagate
Let certbot verify the DNS changes
Use the GitLab API to deploy the generated certificates to GitLab pages

Invoking certbot

In step 1, we call certbot will the preferred method DNS. We also need to supply an email id (used by Let's Encrypt to notify domain expiration). The EMAIL_ID environment variable can be used to store the email id of the user.

Certbot runs in an interactive mode by default. It is not desired in a scripted environment. Certbot also provides mechanisms to deploy the certificates to a local server automatically, but as we are hosting our website on Gitlab, we do not want the automatic deployment facility. So we need to invoke the certbot command with --manual and certonly modes.

certbot   --manual \
          --preferred-challenges dns \
          --agree-tos \
          --email "${EMAIL_ID}" \
          --no-eff-email \
          --expand \
          --renew-by-default \
          --manual-public-ip-logging-ok \
          --noninteractive \
          --redirect \
          --config-dir ${DIR}/generated/config \
          --work-dir ${DIR}/generated/work \
          --logs-dir ${DIR}/generated/logs \
          --manual-auth-hook ${DIR}/auth_hook.sh \
          -d yashagarwal.in \
          certonly

The explanation for most of the flags used in the above command can be found by running the following command -

certbot --help

The --manual-auth-hook flag is worth looking. This hook provides a mechanism to specify the executable, which can be used to facilitate domain ownership validation. In this case, the hook points to a script auth_hook.sh, which then calls a Go client, which interacts with GoDaddy API.

Adding DNS entry to GoDaddy DNS manager

Certbot supplies two environment variables CERTBOT_DOMAIN, which contains the domain name to be verified and CERTBOT_VALIDATION, which includes a random string corresponding to _acme-challenge TXT entry. What this means is that, if I have

CERTBOT_DOMAIN=yashagarwal.in
CERTBOT_VALIDATION=6VNg5kDVI_BF1S9N5s74LTBHQnwDpQqKlblKRjIzBwM

Then the DNS manager should contain a TXT entry _acme-challenge.yashagarwal.in with the value of 6VNg5kDVI_BF1S9N5s74LTBHQnwDpQqKlblKRjIzBwM.

The auth_hook.sh file calls the Go client with the abovementioned environment variables. The relevant code can be found here.

Once all the DNS entries are added, the auth_hook.sh script will sleep for 10 minutes. It is to allow DNS changes to propagate throughout the Internet. It is a random duration as I could not find any GoDaddy support page mentioning the exact period used by them.

Generation of certificates

Once the auth_hook.sh script returns successfully, certbot will verify the DNS records. If the verification is successful, certbot will generate the certificates in ./generated/config/live/{CERTBOT_DOMAIN} directory.

Deploying the certificates to GitLab

I use the following command to deploy the certificates to Gitlab pages where my website is hosted -

curl  -vvv \
      --request PUT \
      --header "Private-Token:${GITLAB_TOKEN}" \
      --form "certificate=@${key_dir}/fullchain.pem" \
      --form "key=@${key_dir}/privkey.pem" \ "https://gitlab.com/api/v4/projects/yashhere%2Fyashhere.gitlab.io/pages/domains/yashagarwal.in"

where

key_dir="./generated/config/live/yashagarwal.in"

Moreover, GITLAB_TOKEN is an environment variable that contains the API token generated from the Gitlab settings page.

Automatic Deployment using Travis CI

It is not automation if I have to run this script manually every three months. So I created a Travis CI job to automate this process. The job will run every month and deploy my certificates automatically. It has been four months, and I have not faced any issues with this setup.

The code for this post can be viewed at Github.

Thanks for reading. Cheers 😄

Setting Up ModSecurity on Ubuntu

Mon, 01 Jul 2019 12:50:18 GMT

Recently, I am experimenting with Web Application Firewalls a lot. ModSecurity is one of them. It is the most famous and useful open-source Web Application Firewall (WAF) in existence. It is supported by various web servers such as Apache, Nginx, and IIS.

The job of ModSecurity is to sit in front of the application web server and check the incoming requests and outgoing responses to filter out malicious content. It does so by the use of powerful and complex regular expressions. ModSecurity uses a rule language for its rules. The rule language has variables and operators defined to aid in the process of parsing HTTP requests.

ModSecurity, in itself, cannot block or allow requests. It is just a rule engine. It requires rules to operate appropriately. That's where its sister project, Core Rule Set (CRS), comes into the picture. CRS is a rule set developed to be used with ModSecurity. It has been in active development for several years now and is very mature. Together, ModSecurity and CRS form a formidable defense against the widespread web application attacks.

Now that you know, what a WAF is, let's proceed to install ModSecurity on Ubuntu. I will be compiling ModSecurity's latest version on Ubuntu 18.04. We will also configure ModSecurity to use Core Rule Set.

Installing Dependencies

ModSecurity requires some dependencies to work correctly. Let's install them -

First, upgrade the Ubuntu system.

sudo apt-get -y update
sudo apt-get -y upgrade

Now install the dependencies.

sudo apt-get -y install git libtool dh-autoreconf pkgconf gawk libcurl4-gnutls-dev libexpat1-dev libpcre3-dev libssl-dev libxml2-dev libyajl-dev zlibc zlib1g-dev libxml2 libpcre++-dev libxml2-dev libgeoip-dev liblmdb-dev lua5.2-dev iputils-ping locales apache2 apache2-dev ca-certificates wget

Optional: clean up the Ubuntu caches.

sudo apt-get clean && sudo rm -rf /var/lib/apt/lists/*

Install SSDeep as well (as done here)

cd ~
git clone https://github.com/ssdeep-project/ssdeep
cd ssdeep
./bootstrap
./configure
make
sudo make install

Compiling ModSecurity

Let's clone ModSecurity from Github.

cd ~
git clone -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
cd ModSecurity
git submodule init
git submodule update
./build.sh
./configure
make                # takes ~8 minutes on AWS t2.micro
sudo make install

Compiling ModSecurity-apache connector

To configure it with Apache, we will require ModSecurity-apache connector. Let's install that as well.

cd ~
git clone https://github.com/SpiderLabs/ModSecurity-apache
cd ModSecurity-apache
./autogen.sh
./configure --with-libmodsecurity=/usr/local/modsecurity
make
sudo make install

Setting up CRS rules

Now, let's download CRS rule set as well.

cd ~
git clone -b v3.2/dev https://github.com/SpiderLabs/owasp-modsecurity-crs
sudo mv owasp-modsecurity-crs/ /usr/local/

Rename CRS configuration file -

sudo mv /usr/local/owasp-modsecurity-crs/crs-setup.conf.example /usr/local/owasp-modsecurity-crs/crs-setup.conf

Setting up ModSecurity

Now, we need to create a file in the Apache modules directory, so that Apache can know, how to activate ModSecurity.

Create /etc/apache2/mods-enabled/security3.conf file and paste the following contents -

LoadModule security3_module /usr/lib/apache2/modules/mod_security3.so
modsecurity on
modsecurity_rules_file '/etc/apache2/modsec/main.conf'

As you can see, the last line in the above code block reference a file main.conf in a folder modsec. This folder will not be present by default. We need to create that.

sudo mkdir -p /etc/apache2/modsec

Setup ModSecurity configuration file -

# enables Unicode support in ModSecurity
sudo wget -P /etc/apache2/modsec/ https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/unicode.mapping
 
sudo wget -P /etc/apache2/modsec/ https://raw.githubusercontent.com/SpiderLabs/ModSecurity/v3/master/modsecurity.conf-recommended
sudo mv /etc/apache2/modsec/modsecurity.conf-recommended /etc/apache2/modsec/modsecurity.conf

Change the SecRuleEngine directive in the configuration to change from the default "detection only" mode to actively dropping malicious traffic.

sudo sed -i 's/SecRuleEngine DetectionOnly/SecRuleEngine On/' /etc/apache2/modsec/modsecurity.conf

Change the location of modsec_audit.log file to Apache log directory.

sudo sed -i 's/SecAuditLog \/var\/log\/modsec_audit.log/SecAuditLog \/var\/log\/apache2\/modsec_audit.log/' /etc/apache2/modsec/modsecurity.conf

To configure ModSecurity to use CRS rule set, put the following text in /etc/apache2/modsec/main.conf file.

Include "/etc/apache2/modsec/modsecurity.conf"
Include "/usr/local/owasp-modsecurity-crs/crs-setup.conf"
Include "/usr/local/owasp-modsecurity-crs/rules/*.conf"

Also enable some Apache modules for better functioning of ModSecurity.

sudo a2enmod unique_id headers rewrite actions dav dav_fs

Now restart the Apache server

sudo systemctl restart apache2

Fixing some common issues

Sometimes, I had encountered errors when ModSecurity was not able to append logs to its log file. I figured out that ModSecurity did not have enough permissions to write that file. We can fix this issue quickly.

First, test if you really have this issue or not.

$ curl 'http://localhost/?q=">
<script>
  alert(1)
</script>
'
<!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
  <head>
    <title>403 Forbidden</title>
  </head>
  <body>
    <h1>Forbidden</h1>
    <p>You dont have permission to access / on this server.<br /></p>
    <hr />
    <address>Apache/2.4.29 (Ubuntu) Server at localhost Port 80</address>
  </body>
</html>

Now go to Apache log directory and check the contents of modsec_audit.log file.

cd /var/log/apache2
tail modsec_audit.log

You should see the following content -

---0LzdyETA---A--
[01/Jul/2019:14:42:41 +0000] 156199216179.666171 127.0.0.1 41824 ip-xxx-xx-xx-xx.ap-south-1.compute.internal 80
---0LzdyETA---B--
GET /?q="><script>alert(1)</script> HTTP/1.1
Host: localhost
User-Agent: curl/7.58.0
Accept: */*
 
---TqjMwy7h---D--
 
---TqjMwy7h---F--
HTTP/1.1 403
 
---TqjMwy7h---H--
ModSecurity: Warning. detected XSS using libinjection. [file "/usr/local/owasp-modsecurity-crs/rules/REQUEST-941-APPLICATION-ATTACK-XSS.conf"] [line "37"] [id "941100"] [rev ""] [msg "XSS Attack Detected via libinjection"] [data "Matched Data: XSS data found within ARGS:q: "><script>alert(1)</script>"] [severity "2"] [ver "OWASP_CRS/3.1.0"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-xss"] [tag "OWASP_CRS/WEB_ATTACK/XSS"] [tag "WASCTC/WASC-8"] [tag "WASCTC/WASC-22"] [tag "OWASP_TOP_10/A3"] [tag "OWASP_AppSensor/IE1"] [tag "CAPEC-242"] [hostname "localhost"] [uri "/"] [unique_id "156198848361.198287"] [ref "v8,27t:utf8toUnicode,t:urlDecodeUni,t:htmlEntityDecode,t:jsDecode,t:cssDecode,t:removeNulls"]
....
....
 
---TqjMwy7h---I--
 
---TqjMwy7h---J--
 
---TqjMwy7h---Z--

If you do not see the following content, and the file is empty or it does not exist, then ModSecurity was not able to open this file for writing. Use the following fix -

# find out the user, Apache is running as
apache_user="$(ps -ef | egrep '(httpd|apache2|apache)' | grep -v `whoami` | grep -v root | head -n1 | awk '{print $1}')"

Add this user to adm group which owns the Apache logs directory in Ubuntu.

sudo usermod -G adm www-data

Now, change the owner of Apache log directory to apache_user.

sudo chown -R $apache_user:$apache_user /var/log/apache2/*

Now, ModSecurity should be able to append logs to the file modsec_audit.log.

Bonus: Enabling JSON logs

Note: Honestly speaking, I was not able to make it work every time. I do not know what is the issue, but it works with some of the installations, and with some of the installations, it just doesn't log anything to the audit directory. If anyone has managed to make it work consistently, please let me know.

Edit (13/07/2020): The JSON logging works fine. The issue was that ModSecurity did not have permission to create subdirectories in the Apache log directory. I suppose it is something related to SELinux. However, a simple solution is to add the user under which the Apache process runs to the adm group. It might not be the right solution security-wise. However, from a quick remediation point of view, it works. Please let me know if you identify any better solution to fix the problem.

Anyway, if you are like me, who do not like the default ModSecurity log format, ModSecurity provides an option to generate logs in JSON format as well. To enable JSON support, the YAJL library should be installed. We already installed this package when we were installing dependencies, so our ModSecurity setup is compiled with JSON support. Let us now configure ModSecurity to generate JSON logs.

Open the /etc/apache2/modsec/modsecurity.conf file and find the following lines -

SecAuditLogType           Serial
SecAuditLog               /var/log/modsec_audit.log

Once you have found the following lines, replace these lines with the following lines

SecAuditLogFormat         JSON
SecAuditLogType           Parallel
SecAuditLog               /var/log/apache2/modsec_audit.log
SecAuditLogStorageDir     /var/log/apache2/audit/
 
SecAuditLogFileMode       0644
SecAuditLogDirMode        0755

Restart Apache server

sudo systemctl restart apache2

Now, go to /var/log/apache2/ directory and create audit folder.

sudo usermod -G adm $apache_user
 
cd /var/log/apache2
sudo mkdir audit
 
# make `apache_user` owner of this directory as well...
sudo chown -R $apache_user:$apache_user /var/log/apache2/audit

Now, ModSecurity should be able to generate JSON logs in this directory. ModSecurity generates logs in the following format -

ubuntu@server:/var/log/apache2$ tree audit
audit
└── 20190701
    ├── 20190701-1132
    │   ├── 20190701-113225-156196094515.868593
    │   └── 20190701-113226-156196094691.154769
    ├── 20190701-1211
    │   ├── 20190701-121122-156196328239.048942
    │   └── 20190701-121122-156196328243.018882
 
    ....
    ....

Now, your site should be relatively more secure than before.

A warning, though

CRS is known to generate a lot of false-positive when enabled completely. We have not touched CRS paranoia levels. By default, it is set to paranoia level 1, which is known to produce false positives rarely, but still, as a measure of precaution, monitor your site's traffic for some time, and then decide if you need to disable some of the CRS rules or not.

Travelogue - Chikmagalur

Sat, 29 Jun 2019 00:40:03 GMT

South India is blessed with nature throughout the year, but monsoons bring a different flavor and ethereal quality to some places. Although the rainy season might not be the best time to travel, there is a certain charm in holidaying during the downpour.

We (I and my flatmates in Bengaluru) were planning a trip to Chikmagalur since the beginning of June. One of my friends knows driving, so we decided to rent a car and drive ourselves to a distance of approximately 250 KMs. We booked the car through Drivezy. They had promised a Mahindra KUV100, but at the last moment, they couldn't keep their commitment and gave us a Honda Amaze with automatic transmission. My friend who was driving had never driven such a car, so we all got scared of our fate for the next two days, but we decided to go ahead with it, trusting our friend.

I too know driving, but like every other aspect of my life, I never gained enough confidence to go on such a long drive. It was a similar case this time as well.

The way to Chikmagalur from Bengaluru is the well-maintained NH75 (it has a total of four tolls, so maintenance is expected). We started early on the morning of Saturday, 22nd June to avoid Bengaluru's traffic. Nature was accommodating that day, and the Sun 🌞 stayed hidden behind clouds for most of our journey. That made our 6-hour drive very enjoyable.

We were not in any hurry and wanted to enjoy every moment of our trip to the fullest. With Punjabi beats playing in our car, we stopped at various places on the highway.

Chikmagalur means "The town of the younger daughter" in the Kannada. The town is said to have been given as a dowry to the younger daughter of Rukmangada, the legendary chief of Sakkarepatna and hence the name.

We stopped at Adyar Ananda Bhavan 🏨 for breakfast. I have tasted sweets from A2B, and the taste was the best that I could get in the South. So I decided to eat Masala Dosai and Idli Sambhar there. The food did not disappoint me at all.

After breakfast, we resumed our journey. The first stop in our trip was the gorgeous Sri Chennakeshava Temple. Chenna means beautiful and Keshava refers to Lord Vishnu. That should give you an idea, the presiding deity of the temple is the Handsome Vishnu.

The whole premises of the temple is filled with sheer poetry on the walls in the form of stone sculptures. I was completely mesmerized by its intricate but aesthetically pleasing carvings.

On the entrance of the temple, a vast golden Gopuram welcomes the devotees. I was amazed by the level of details on the sculptures engraved on the Gopuram. My phone's camera will not be able to do justice to this intricate artistry.

The one specialty of the main temple is that it does not feature a dome. I have not seen any major Hindu temple without a dome over the idol of the main deity. It seems like an aberration to me, but a welcoming one. A disclaimer, I am not an architecture buff, so I do not know how correct my facts are here.

The inside of the temple is pretty dark. There are several pillars inside the temple. Each of the pillars is decorated with various types of sculptors.

However, the fascinating thing is at the ceiling. The carvings done on the ceiling of the temple is just amazing.

As I mentioned above, I am not an architecture buff, so I would not be able to write a lot about the architecture of the temple. However, enjoy some more photos.

After this temple, we continued our journey to Chikmagalur. We had booked a homestay option. The name of the homestay was Sunshine Palm Retreat, if I recall correctly. It was one of the best stays I have experienced on any trip till now. It was not crowded, the staff was co-operative, and the ambiance was excellent.

On the evening of Saturday, we decided to go for our second destination of the day - the so-called shooting point. When we searched on Google maps, we found one location with a similar name. We started driving in that direction. However, it turned out that we were going to the wrong place. We tried asking locals, but language was an issue. Most of the signboards are also in Kannada only. Somehow we ended up getting on the right track and reached the correct location on time. After reaching there, Google Maps shows that the place is called Bagmane Sunrise Point. Even this name does not seem to be right.

As soon we reached there, I realized that all the efforts that we put in reaching there were worthwhile. It was an open ground area on top of a small hill at one end of a small village. I started roaming here and there, enjoying the serenity of nature. I am not a very photogenic person, so I prefer to click pictures of objects and nature more than standing in front of a camera at different angles.

We spent around 1.5 hours there and then came back. India vs. Afghanistan cricket match was live on TV, and we fixed ourselves in front of the TV for the next 4 hours. India somehow won that match. Thank God!

The next day, it was raining from the early morning. We did not want to take any risk by driving on the steep and narrow zig-zag roads. So we dropped our plan to go to Butter Milk Waterfalls and instead decided to go to Mullayanagiri peaks as our last destination. We again lost our way, but somehow managed to reach the place.

It was drizzling there. However, we were at such a high altitude that we had to bring our sweaters out. Surprisingly, I found vendors selling Maggi there. I was not expecting the taste to be good, but I was wrong. For that sort of place, the Maggi was delicious.

https://www.youtube.com/watch?v=F9SpC8nFF0A

Despite the weather conditions, we decided to climb to the top of the hill. As we were going up, the weather was turning more and bleaker. Out of five of us, only three managed to reach the top, including me. The view from the top of the hill was just amazing. Wherever I saw, I saw just the mountain mist. My clothes were wet, and my spectacles were completely covered in fog.

I shot a video when we were coming down from the hilltop.

https://www.youtube.com/watch?v=So_UD2tpRic

To summarise, the timing of this trip was not appropriate, but I do not regret it. I have gone to other hill stations after the monsoons get over, but it is a different experience altogether to go in a rainy season. Chikmagalur is worth more than one visit. Do try to go at an appropriate time if you want to visit more places.

Thanks for reading. Cheers 😄

Go + gRPC + OPA - A Perfect Union - Part 3

Mon, 18 Feb 2019 01:17:15 GMT

I finished my last post with the following issue -

Now, here one problem arises, how to make sure that the search results will not return any book which the user is not authorized to access. We will solve this problem using OPA in the next and last post of this series.

Let's solve this issue now. We will use OPA's declarative language, Rego, to implement policies which will decide on the basis of some user-provided data, which all objects are to be returned to the user.

We will also define a list of all the users who are part of this library. Here we are hardcoding this data, as I did not want to waste my time in implementing a user registration service, but this functionality is not very important from our point of view. We will require only one field from this users data - the user_type field. This field will determine what the access level for the user is. We have already added the access_level field in the Book definition of our proto file.

When the user wants to search for a particular book, it will provide its user_type the ISBN of the book to our service. Our service will take that ISBN and pass it to the OPA server. OPA server already has the Book data and the User data. Now it has the required ISBN to query the Book data. The Rego policy will query the Book data by ISBN. It will also check for the access_level condition. Moreover, after this operation, it will return the resultant set of books that satisfy both the requirements.

Here is the Rego policy -

package library
 
import data.books
import data.users
import input
 
search_books[book] {
  input.book.isbn:= books[i].isbn
  input.user.user_type >= books[i].access_level
  book: books[i]
}
 
list_all_books[books[i]] {
  input.user.user_type >= books[i].access_level
}

The user data is here and the book data is here.

A sample input request is shown below -

{
  "input": {
    "book": {
      "isbn": "1128959038"
    },
    "user": {
      "user_type": 3
    }
  }
}

The input is the data that the user is providing. In search_books function, the input ISBN is matched with the ISBN of all books one by one. Then the resultant set of books is filtered by user_type and access_level (these two fields are essentially the same). In the last, the resultant set of books is assigned to the variable book which will be returned to the gRPC service.

The list_all_books function is implemented similarly. The only difference is that we do not need to filter the books by ISBN. Filtering by access_level is enough.

Now our library service is completed. It is a very basic service. The intention was to show that the decision-making process can be offloaded to the OPA to reduce the complexity of the services. In this example, the advantages might not be obvious, but in large production environments, where many services are running, it can make a significant difference.

The code for this series can be found on my Github account.

I hope you liked the article. Share your views and suggestions in the comments.

Thanks for reading. Cheers :)

Go + gRPC + OPA - A Perfect Union - Part 2

Sun, 17 Feb 2019 09:14:56 GMT

In the last post, we discussed about the structure of our library application. In this post, we will define the data definitions using protobuf, and then we will use these definitions to create a Go service. We will also add a REST interface to the service. So let's get started.

Defining Proto Definitions

gRPC uses protocol buffers for serializing structured data. To define the structure of the data that you want to serialize, we use a proto file - it is a simple text file that contains all the logical pieces of your data in the form of messages, and the methods that will be called over the network. To know more about the syntax of proto files, visit this link.

I have defined the following proto file -

syntax: "proto3";
package library;
import "google/api/annotations.proto";
 
service LibraryService {
  rpc ListAllBooks(QueryFormat) returns (Books) {
    option (google.api.http): {
      post : "/listBooks"
      body : "*"
    };
  };
  rpc AddBook(QueryFormat) returns (Response) {
    option (google.api.http): {
      post : "/addBook"
      body : "*"
    };
  };
  rpc SearchBook(QueryFormat) returns (Response) {
    option (google.api.http): {
      post : "/searchBook"
      body : "*"
    };
  };
}
 
// the library
message Library { Books books: 1; }
 
message Books { repeated Book books: 1; }
 
// metadata about a book
message Book {
  string title: 1;
  string author: 2;
  string isbn: 3;
  int32 no_of_copies: 5;
  int32 access_level: 6;
}
 
// details about a user
message User {
  enum UserType {
    // https://github.com/golang/protobuf/issues/258
    GARBAGE: 0;
    Student: 1;
    Administration: 2;
    Faculty: 3;
  }
 
  string name: 1;
  int32 id_no: 2;
  UserType user_type: 4;
}
 
message QueryFormat {
  Book book: 1;
  User user: 2;
}
 
message Response {
  string action: 1;
  int32 status: 2;
  string message: 3;
  oneof value {
    Book book: 4;
    User user_data: 5;
  }
}
 
message Empty {}

To compile it, run the following commands -

protoc -I/usr/local/include -I. \
-I$GOPATH/src \
-I$GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \
--go_out=plugins=grpc:. \
api/library.proto

protoc -I/usr/local/include -I. \
  -I$GOPATH/src \
  -I$GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \
  --grpc-gateway_out=logtostderr=true:. \
  api/library.proto

It will generate corresponding Golang definitions of the messages and services defined in the Proto file. These definitions can be used by the server and client stubs to communicate with each other.

Implementation of Go service

Now we can start implementing the code for our services AddBook(), ListAllBooks() and SearchBook(). It is going to be a very naive implementation of a library system, but it will be sufficient to learn all the concepts.

My implementation of the server stub is hosted here. A basic flow diagram of this implementation will look like this -

The gRPC server will listen on port :50051, and a REST HTTP server will listen on port :8181. The OPA server is running on port :8182. The REST server is implemented using gRPC-Gateway. There are three methods - AddBook(), ListAllBooks(), and SearchBook(). These methods can be called using either gRPC methods or using the REST endpoints /addBook, /listBooks and /searchBook. By design, the library gRPC service will not implement the authentication part of the service. The main purpose of using gRPC here is to provide a scalable and secure medium where all the communication between client and server is happening in binary format, which is slightly more secure than the traditional mediums. In the current form, this gRPC server will accept requests from everyone and execute the desired functions. That is not desirable. What if a student tries to add a book to the library. Only Admins should be allowed to execute such functions. What if someone who is not a student of the University tries to access the service. How to stop them?

There are two steps to solve this issue -

Authentication - It mainly deals with the question - who are you? It is a way to gain access to the system by verifying your identity. In our case, a user will provide its username and password to access the library service. Without this authentication, the user will not be able to access the system. We will not be implementing authentication functionality in our application.
Authorization - It deals with the question - which resources are you allowed to use? OPA can be used here to define various rights based on the access levels of the users.

If you have noticed, I have defined an access_level field in the proto definition of the Book. This field will tell us what is the minimum access level required for a user to access this book.

Again, in the proto definition of the User, I have defined a user_type field. This field will serve as an indicator of the access rights of the user. In the real world, these access rights will be decided after the user has authenticated herself to the system, but here, we will hardcode the access rights.

So, only users with access rights equal to Administration will be allowed to add books to the system. Here we do not care who the user is. If the user is supplying the correct access right, she will be allowed to operate. The authentication logic in real-world scenarios will determine the who part.

There are some books in the library, which have access rights equal to that of a Faculty. It means that only faculties will be allowed to access those books. The students will not be able to access these books, even while searching for books using ISBN. This kind of mechanism can be implemented using OPA very quickly. We will see the implementation of the OPA part in the next post.

While querying the service, users are required to supply their identity (at least user_type) and the book ISBN if they are searching for some book. The administrators are supposed to provide the name, author, access level, number of copies, and ISBN while adding the books. I have not added the error checking functionality in the code, but it should be easy enough to implement such functionality.

The main.go file is the starting point of this service. It will spawn two servers in two Go Routines. Ideally, some synchronization mechanisms should be implemented in the code to avoid race conditions in some cases - for example, what will happen if two or more clients are trying to add the same book simultaneously. Here in our case, nothing serious will happen, as OPA will take only one book per ISBN, and discard all the other books with the same ISBN even if the other metadata is different (I designed the service in this way to keep the code easy enough to understand), but if there are other operations like DeleteBook and IssueBook, then the race conditions can cause issues.

In the AddBook() function, the user provided book details will be sent to the OPA server using a REST call. OPA will store this information in its in-memory store at a unique place determined by the ISBN of the book. In actual cases, the data should be stored in some persistent storage, such as a DB. OPA will take the information from the DB. Again, to keep the implementation easy enough to understand, we are not using any such persistent storage. If any other book with different metadata but the same ISBN comes, then OPA will overwrite the existing book with the new one.

In the SearchBook() function, the user will provide the ISBN of the desired book. The gRPC service will call the OPA using REST API and find if any such book exists or not.

The ListAllBooks() is different in the way that it does not need any ISBN.

Now, here one problem arises, how to make sure that the search results will not return any book which the user is not authorized to access. We will solve this problem using OPA in the next and last post of this series.

I hope that this post was helpful. If you have any doubts or want to say anything else, please comment. It will be a great motivation and appreciation for me.

Thanks for reading. Cheers 😄

Go + gRPC + OPA - A Perfect Union - Part 1

Sun, 10 Feb 2019 08:59:08 GMT

TL;DR – In a series of blog posts, I will be implementing a simple library application supporting both gRPC and REST interfaces using Go, gRPC, and OPA. My approach might not be the most optimal one, but I am learning these technologies currently. Please give your valuable suggestions and be kind :)

I have been learning the basics of microservices and Golang lately. On the work front, I got a chance to work on Go, gRPC, and Open Policy Agent as my first professional project. In this post, I will be demonstrating what I learned in the last few months. We will be implementing a simple gRPC based library service, which will be able to serve requests using both gRPC and REST calls. It will also incorporate the Open Policy Agent (OPA) to provide the authorization to users. Let's begin with a quick introduction to gRPC and OPA.

gRPC

gRPC is Google's implementation for Remote Procedure Calls(RPC). RPC is mainly used in building scalable distributed systems. While REST has a limited set of verbs, RPC can define any function calls, including synchronous and asynchronous calls.

In gRPC, the client can make procedure calls as if the requests are made to some local function. However, the underlying client stub (auto-generated) will send the call to the server. The server will have a similar server stub, which will be able to handle the requests coming from the client. The server will send the response to the client using similar mechanisms over the network. All the communication is serialized to binary format, so it is ideal for distributed systems as binary format tends to be on the faster side for large amounts of data.

For more info about gRPC, visit the official website.

Open Policy Agent (OPA)

OPA gives us the ability to define a fine-grained policy control mechanism. However, I think the most critical benefit of using OPA is that it gives you the ability to decouple your services and the definition of policies from the enforcement of it.

There are mainly two parts of OPA -

A JSON document store where you can define anything from your users, access roles, permission levels, etc.
A policy is written in a declarative language. This policy gives you new derivative data from the original JSON document store evaluated by the policy. The declarative language is called Rego, and these policies are also documents that generate results according to the defined policy. The users query these results.

This info will be sufficient for our use case. More info about it can be found at the official website.

The skeleton of our Application

We will be building a command-line library application. This application will support adding, deleting, searching, issuing, and returning of books. There are three types of users – student, faculties, and staff. Not all users have a similar kind of access level. There are some books which are only reserved for students, and some are exclusively reserved for faculties and so on.

I think emulating the library at Hogwarts will be a good idea here. Students and ordinary people were not allowed inside the restricted section of the library. We will emulate that restricted section using the authorization mechanisms provided by OPA.

The gRPC protocol will handle the communications part in our app, but not everyone in Hogwarts is using gRPC. Wizards are still in love with REST (not a bad thing, though), so we will provide them alike the REST interface to interact with our service. We do not want to face the wrath of the wizards, after all.

OPA can either be used as a standalone application or be embedded in the Go service as well. Both approaches have their pros and cons. I have decided to use OPA as an independent service, as it will be more comfortable that way for us to push authorization data and policies to it if such a need arises in the future.

One more issue remains - where to store the added books. Shall we persist them? In practical scenarios, persistence is always a good idea, but here I do not want to complicate things too much. We could use any Object storage servers (e.g., Minio) to store the JSON data generated from our gRPC methods, but that will unnecessarily add complexity to our simple scenario. Wizards do not like complicate stuff, you know!

In the next posts, we will define our gRPC proto file and build the Go service around it. Then we will add a REST interface to our service for simple CRUD operations in our application. Then we will add OPA authorization to restrict our users from adding and viewing books which they are not supposed to access.

Thank you for reading. Cheers 😄

2018 - Year in Review

Mon, 31 Dec 2018 16:32:08 GMT

The year 2018 has been quite a roller coaster year for me. I saw many ups and downs, many successes, and many failures faced many challenges on the personal front and struggled to keep myself on track. I have been working hard on improving my lifestyle (not materialistically) for the last few years, and the benefits have begun to show up now. This year was a pinnacle in that aspect. A lot of micro changes happened in my life this year.

I often read my blog posts from last year. Reading these posts give me a good feeling about the improvements that I could bring in myself during the previous three years. Writing year reviews and posts like this, this and this serve as documents of my deeds from past for my future. It helps me in not repeating the same mistakes. As a human being, I cannot run away from committing mistakes, but I can always make sure that I do not repeat the same mistakes.

Those who cannot remember the past are condemned to repeat it.

Blogging & Writing

I had set a target to write 15 blog articles at the beginning of this year. In the end, I managed to achieve this goal. I managed to write 17 blog posts and six book reviews. My blog managed to get much traffic this year, mainly thanks to some of my technical posts.

I got some good responses from people about my technical blog posts. My aim to start writing technical posts was to keep an account of what I learn. It feels very nice when I go back to my posts to read how I did something last time. So I consider myself successful that at least I could benefit from my writing.

Though, in 2018, after leaving college, the frequency of technical posts on this blog has dipped to almost zero. It is probably because I am not exploring enough after joining the corporate world. In college, I used Linux all the time, but now it is nearly four months since I last booted Linux in my system. I think it is yet another phase of my learning, and I will take some time adjusting with this. I have some lovely technical post ideas, and I hope that I will be able to articulate them soon next year.

Reading

For most of the year, I did not pay any heed to books, and I was never very interested in buying books. However, at the end of October, I purchased a Kindle Paperwhite. I consider this to be the best investment done by me this year. Since buying it, I have read eight books. I finally managed to finish my Goodreads reading challenge of 10 books. Although I could not finish one book and included one audiobook to this challenge to complete my target on time.

This year, I discovered my interest in Economics. I am reading some economics-related literature lately. It is a subtle shift from my all-CS life. I plan to read a lot of economics-related books in 2019 to gain more expertise in this domain. But I still need to figure out a way to stop it from interfering with my first (or second, maybe!) love.

Some of my favorite reads from this year (not in any particular order) -

Misc

I used Instapaper extensively throughout the year to manage and save the articles that I liked. Although I was never very consistent in reading those saved articles, it was still a good exercise. I finished all my pending pile of saved articles by the end of the year. Moreover, for this post, it became straightforward for me to recollect those links. This list grew excessively large, but I did not want to lose any of these links because of some random third-party service shutting down, so I am adding them here for easy reference in the future.

Next year onwards, I plan to read one academic paper from my favorite areas of CS in every two weeks and summarise them to improve my understanding of the domain (the /readings section of my blog was initially intended for this purpose). Moreover, seeing my experience in the last two months, I feel encouraged to increase my Goodreads reading challenge to 20 books.

Work

The year 2018 was a pivotal year for me on this front. It was the year when I had to come out of the comfort zone of my parents' protection. My professional life began this year, and I started earning MONEY. It feels delightful when you see the money gets deposited to your bank account every month for the hard(!) work that you have done for the last one month. But

With great power comes great responsibility.

I believe that I have been able to control this feeling of power quite nicely until now. Probably being an Agarwal helped :P

Coming back to work, those who do not know, I joined Cisco Systems this year as a Software Engineer. The job is pretty good for my initial expectations. I made some good friends. First time in my life, I managed to make a group of friends.

Quite an achievement.

Studies

This year was officially my last year in formal education.

Although I failed to finish my bachelors with a satisfactory final GPA (at least I am not happy with it), I am very much satisfied with all my learnings from college. I learned many life lessons, kept myself in check, built some useful contacts, learned a lot about many CS-related things, and ended up with a decent job in the end.

However, I sense that this is not the happy ending of my journey towards learning, and I have a lot more to learn. I recently read the book "The Alchemist," and the central theme of this book is to keep looking for your legend. I am still searching for my legend. In 2019, I expect to find new ways to keep learning and explore new areas both in CS and other fields of life.

Relationships

Before this year, I had an assumption that I lack social skills. I hesitated in talking to people. I decided to change it this year. I was waiting for a change in my life, probably because NITC had become too monotonous for me. It was one of my new year's resolution that I will change myself and become more approachable to people.

I do not know if I have changed or not, but I feel much more confident now than ever in communicating with people. I hope I will continue to improve in 2019 as well.

I was trying to get rid of Facebook for quite some time. This year, I finally managed to control my addiction. For a good part of the year, either my Facebook account remained deactivated, or I was logged out of it. In the past, I used to feel a strong urge to open Facebook every ten minutes, but this year, I was able to keep this urge in control. When my account was not deactivated, I kept myself so busy in other works, that I did not even think of Facebook.

Moreover, the good thing is that I did not notice any symptoms of withdrawal symptoms because of this sudden quitting of Facebook. A friend's advice worked nicely. It was quite tricky, indeed.

I was never addicted to any other social media platform, but I decided that I can use Twitter for my benefit. I feel Twitter is a useful tool if utilized correctly. I followed many people who tweet regularly about CS related stuff. I open Twitter once every 2-3 days for a very short duration to collect some articles and tips. Nowadays, these articles, along with a few blog articles from Feedly, mainly form my Instapaper feed. Though Instapaper recently switched to commercial model again, I decided to keep using it for the time being until I find some other alternative.

Miscellaneous

I had decided to exercise for at least 90% part of 2018. I failed miserably. Leave alone every day; I could not even do it continuously for one month. I need to make this a top priority. Otherwise, it is going to create trouble for me soon.

For the last two years, I am trying to get up early in the morning every day. My body does not seem to care, though. It seems a classic example of the struggle between mind and body. My brain says that I am an early bird, but my body straightforwardly rejects this idea. I have felt the positives of waking up early, so I am still trying my best so that my body can adjust to this change. I will keep working on this habit in 2019 as well.

I noticed a bad habit in myself - procrastination. I procrastinated a lot in 2018. Things that I could have finished well within the deadlines took me ages to complete. I was always late in my BTech project deadlines, I could not finish one of my hobby projects, I could not finish my study targets on time, and many other such examples haunted me this year. It is one of my most important goals of 2019 that I need a way to find how to concentrate on one thing at a time. Many things distract me while I am working. Technology itself is a significant distraction, but because I am a CS professional, this is one thing from which I cannot run away. However, I will still try to avoid getting distracted by my curiosity and put more focus on doing work.

I am writing more than ever. However, with my increased writing, my English vocabulary is not enough anymore. This year, it happened many times that I wanted to write something, but because of the lack of proper words in my vocabulary, I had to settle with inferior words. I tried multiple times this year to work on improving my vocabulary, but I failed badly. I will increase my focus on improving my knowledge of English in 2019 so that such small issues do not stop me from adding something to my skills.

This year, I experimented and tried to organize myself with to-do lists. It was a fruitful exercise, and it had a lot of sound effects on my productivity. The days when I made a to-do list, I felt very energetic and encouraged to finish all the tasks on my list. However, then I noticed one thing, I could not continue doing this for more than a few days at once. For initial days, I felt encouraged to complete my tasks, but soon it began to appear as a burden, and I felt exhausted. So it seems that something in the middle will work for me. This year, I will continue using to-do lists (I use Google Keep for this) but for short durations of time. I will refrain from using it every day to avoid getting exhausted. I am considering a switch to Notion to maintain my monthly and long-term goals. Probably that will also help me in keeping track of my goals without feeling the heat.

All of my friends love to travel. It seems a common habit in most of my friends. However, I am an exception. I'm not particularly eager to go on trips. I attribute it to my issues with motion sickness. For the whole year of 2018, I do not have a single trip memory other than my travels to Bengaluru and Jaipur airports. Oh yeah, after thinking for some time, I recalled that I went on one trip from Bengaluru to Calicut to attend my convocation ceremony. That's all! I will consider accepting trip invitations from my friends in 2019. Up until now, I keep refusing to join them in their trips due to my health issues, my desire to stay in solitude and partially because of my unwillingness to explore anything outside technology. However, it is time I start changing myself.

It was a grave mistake that during my four-year stay in Kerala, I did not try to learn Malayalam. I feel that I missed an excellent opportunity to learn a new language. Although I am as bad as other north Indians when it comes to knowing multiple languages (correct me if I am wrong, I do not intend to generalize all north Indians), I think that learning a new language can be very beneficial for some parts of my brain. Now I am in Karnataka, yet another south Indian state and with yet another language. It is an excellent opportunity, and I am not going to repeat my mistake this time. Even if I do not succeed in learning Kannada, it will provide enough challenges to my brain for the short term. Who knows, it might also give me a new skill!

Welcome 2019, I hope that you will help me in taking a step closer to discovering myself. Give me new experiences, keep me grounded and help me in contributing to the progress of my India. 😊

Beginning a New Journey

Sat, 25 Aug 2018 15:46:41 GMT

When I was in eighth (or was it ninth?) class, one of our teachers asked in the class, what do you want to become in your life? People said a lot of things. When my turn came up, I stood up and said, "Mein software engineer banana chahta hu" (I want to become a software engineer). It was not that I was genuinely interested in Computers. When I look back now, I realize that it was my cousins who affected my choice (in an indirect way), and their achievements fascinated me to aim at this particular career choice.

My whole life for the last seven years was revolving around this one dream(!) of getting a job in an MNC. I got my chance when I got placed in Cisco last Monsoon. I was super excited about joining Cisco since then. Perhaps because of the MacBook on which I am typing this article right now 😄 and partly because of the monotonicity of the NIT-C environment. I wanted to break out of the comfort zone that I had built there around myself in NIT-C. Changing place seemed the only viable option.

I have been doing a self-introspection for the last four to five months. My blog posts from the previous few months reflect some hints about my introspections. I have mentioned my observations here and there in a few of my blog posts. I discovered some unique aspects of my personality. On the other hand, I found out some traits which are pretty weird (in my opinion :) ). I am not going to publish all of my findings, but one thing became apparent, my social anxiety requires my immediate attention.

It is quite difficult to change some aspects of our personality; usually those we are born with. But most of the qualities that define someone keeps changing. Life experiences play an essential role in determining a person's character. Success and failure define a person. I firmly believe that no person ever remains the same. People continuously go through phases of realizations and the ups and downs in the career, academics, broken trusts, dampening creativity, growing age, success, glory, hatred, and betrayals. People become what they are because of the world around them. These realizations come from experiences and experiences that come from bad experiences.

I learned this the hard way!

So, I decided, I will not commit those similar set of mistakes again. I will make new mistakes 😜. I realized that there exists a world outside computers and tech. I need to go and talk to people. If not random strangers, I should talk to my friends, take part in various activities, play some sport, focus on my fitness and mental health, and go on trips. And of course, perform well in work and maintain a good work-life balance if it is not a myth.

I cannot change the past now. It is only the future that is yet in my power. I am eager to see how my current actions are going to affect my future.

At last, let me finish this article by quoting Winston Churchill (Although I am not a big fan of him, this quote is worth sharing).

If we open a quarrel between past and present, we shall find that we have lost the future.

Cheers. 😄

The Good and Bad about CSED of NIT-C

Thu, 14 Jun 2018 12:29:28 GMT

Four years ago, in this very month of June, JEE Mains results were out. I had screwed up my JEE Advanced, so I had to settle for an NIT. My AIR was good enough to get me a seat of Computer Science and Engineering in any of the NITs except the first three (at that time Trichy, Warangal, and Surathkal). I was skeptical about going to NIT Calicut because of its distance from my native place, so NIT Allahabad was the most suitable choice. However, in the end, I decided to give preference to NIT Calicut over NIT Allahabad. One of the many reasons for this decision was an excellent article by Dr. Dheeraj Sanghi, a retired professor from IIT Kanpur. His blog post was the primary reason; I decided to go with NIT Calicut. Now that I have graduated from NIT Calicut, I think it is the right time to document my observations about the CSE department of NIT Calicut from the perspective of a student.

My exposure to the CSE department was limited to the facilities provided to undergraduate students only (almost everything except some academic facilities), so I am not entirely sure if this article will help any of the postgraduate students out there. However, some points below might help you decide about the non-academic activities of the department.

Faculties

Like every other NIT, NIT Calicut also suffers from the lack of permanent faculties. Most of the faculties in the department are recruited on a contract basis (ad-hoc). It is not necessarily a bad thing, but many a time, an ad-hoc faculty gets to teach a subject where one does not hold strong command. In that case, the subject becomes a burden for both student and teacher. It happened to me many times when I was stuck in a situation that I ended up losing my interest in the subject as well as grades.

Most of the core subjects are taught by permanent faculties who are very good at their subjects. These faculties are very experienced and are alumni of some of India's top IITs and IISc. The department used to ask new hires to get a postgraduate degree from some IIT or IISc, but this does not seem to be the case anymore.

Most of the professors prefer to teach traditional computer science subjects, so if you are interested in AI, Data Science, or any other such areas of computer science, then you might have to depend on self-study.

Club Activities

CSEA is the main departmental club that organizes almost every co-curricular activity in the department. In my opinion, CSEA is the best department association in NIT Calicut. I was not a member of CSEA, so all of my views are from the perspective of an outsider. CSEA organizes talks and workshops for the freshers from time to time. I attended all the lectures and seminars arranged by CSEA in my initial years, and I feel that those workshops were quite beneficial for me as a student.

FOSSCell is a group of students who contribute to open-source software. FOSSCell organizes workshops related to Linux and open source contributions. It also organizes FOSSMeet, the yearly free and open-source software conference organized in NIT-C. It is one of the largest gatherings of open source enthusiasts in South India. I have written two posts about my experience as a part of the organizing team of FOSSMeet. You can read them here and here. Attending and organizing FOSSMeet is one of the best experiences of my college life. The current executives of FOSSCell are actively working on improving the state of open source contributions in NIT-C. I hope the situation will change in the coming years.

The students manage all the labs in the department. The Software Systems Lab is assigned to undergraduate students. This lab is probably the crown jewel and pride of every CS student of NIT Calicut. It remains open most of the time. All the department servers are kept in this lab and are managed independently by the student administrators chosen by the student administrators. The student administrators are given the responsibility to maintain all the department websites. I do not know if any other university or college in India provides such independence to its students. I was one of the student administrators in this lab, and I consider that this was the best thing that happened to me during my graduation.

The Infrastructure

A traditional computer science course does not require many types of equipment except a laptop and a fast enough internet connection. Internet speed is not extraordinary in NIT-C, but it was sufficient for me. The network administrators blocked many websites, but if the site is related to coursework, sending a short mail was enough in most cases to get it unblocked.

The labs are equipped with modern computer systems, and Ubuntu is installed on almost every department system. The particular emphasis on the use of open-source software is a big plus point for the department.

Courses

The curriculum was last updated in 2010. Since then, six batches have graduated. I think it is an excellent time to consider revising it. The current curriculum is good enough for most traditional CS subjects, but its focus on the present buzzwords of the CS world is very less. I heard that the department changed the curriculum for the batches joining 2017 onwards, but I doubt that there are any significant changes.

The academics are taken quite seriously in the department (apparently). The level of question papers in examinations is generally very tough. I always felt the heat, at least. However, teachers are quite friendly and will help you with any of your queries.

The situation of lab courses (except OS, Compilers, and DSA lab) in the department is very pathetic. Operating Systems lab and Compiler lab have a well-defined structure, and the feeling of developing your tiny OS or compiler is one of the best feelings ever. Data Structure and Algorithms lab runs in parallel to the corresponding theory course, and that helps students in understanding theory by practice.

I cannot say the same about any other lab in the department; most of the students end up learning nothing new from these lab courses. The problem is in the way these courses are handled. There does not seem to be a proper structure of teaching in these courses. Students are expected to learn on their own and come to the lab and give exams. Although most of the time, the corresponding theory course carries marks for a mini project which compensates for the lab course, these lab courses do have some scope for improvements.

Placements & Higher Studies

The department does not handle placements, but the quality of education and the focus on industry-oriented courses do affect the placements. The placement department of NIT-C has been doing an excellent job of maintaining a track record of over 90% placements from computer science every year.

Department, as such, does not focus on placements much. I feel that the department is more interested in making researchers than software engineers. Most of the courses are very heavily inclined towards theory. From my batch, I know at least ten people who are planning to go to the USA for higher studies in the next two years. The faculties also encourage students to pursue higher education (and subsequently research). Perhaps, it is one of the reasons that the state of competitive coding is not very good in the department.

Conclusion

Despite continuous lack of good permanent faculties and its location disadvantages, CSED of NIT-C has managed to produce competent engineers year by year. Spending four years in Kerala can be a different experience for outsiders, especially North Indians, but it is an experience worth gaining. If you have a good rank in JEE Mains and do not mind going too far from home, CSED of NIT Calicut is an excellent choice. I hope you will not regret the decision, though I will not say the same about other departments of NIT-C.

If you decide to join NIT-C, do read my article about my experience at NIT Calicut.

Thanks to my friends Aashish, Mahaveer and Faris for reading the drafts of this article and providing their suggestions.

Best wishes. Cheers 😄

Battery Notifications in i3

Tue, 12 Jun 2018 04:15:46 GMT

I am using i3 window manager for the last seven months, and it has been a pleasant and productive experience so far. There were a few hiccups here and there, but that is expected with such minimalistic setups. One thing that I never noticed was the lack of notifications on critical battery levels. For the last few months, my laptop battery was discharging to 0% all the time. Probably this proved to be too fatal for my battery. According to this article, lithium-ion batteries are not expected to go from 100% to 0% frequently. I recently bought a new battery, and I did not want to reduce the lifespan of this battery too. So I decided to set up battery notifications for my i3 setup.

I found a bash script which shows a notification using notify-send when battery charge level reaches or drops below a configured threshold. However, I had to do some additional steps to make this script work on my system.

The first issue was the lockfile program, which was not installed in my system. I installed it using the following command.

sudo apt install procmail

The second issue was more difficult to solve. I planned to set up the script to run every minute using cron. However, it turns out that cron operates in a very minimalistic environment and notify-send requires the presence of some special variables in the environment. These variables are DBUS_SESSION_BUS_ADDRESS, XAUTHORITY and DISPLAY. To provide the values of these variables to the cron environment, I modified the script and sourced a new file .bat_envs.

#!/usr/bin/env bash
 
. /home/yash/.bat_envs
 
THRESHOLD=15
 
lock_path='/tmp/battery.lock'
 
lockfile -r 0 $lock_path 2>/dev/null || exit
 
acpi_path=$(find /sys/class/power_supply/ -name 'BAT*' | head -1)
charge_now=$(cat "$acpi_path/charge_now")
charge_full=$(cat "$acpi_path/charge_full")
charge_status=$(cat "$acpi_path/status")
charge_percent=$(printf '%.0f' $(echo "$charge_now / $charge_full * 100"
 | bc -l))
message="Battery running critically low at $charge_percent%!"
 
if [[ $charge_status == 'Discharging' ]] && [[ $charge_percent -le $THRE
SHOLD ]]; then
  /usr/bin/notify-send -u critical "Low battery" "$message"
 
  current_date_time="`date +%Y%m%d%H%M%S`";
  echo "[BATTERY LOG] = $charge_percent% on $current_date_time"
fi
 
rm -f $lock_path

Read this blog post to understand how this script works.

As the notify-send requires some special X session environmental variables, we will need a method to provide these variables to notify-send in cron environment. The safest way to get X session related environmental variables is to get them from the environment of a process of the user who is logged on to X. The following script will run every time a user logs in and stores these variables in a file .bat_envs.

#!/usr/bin/env bash
 
env_path="$HOME/.bat_envs"
 
rm -f "${env_path}"
touch "${env_path}"
 
copy_envs="XAUTHORITY DISPLAY DBUS_SESSION_BUS_ADDRESS"
 
for env_name in $copy_envs
do
    env | grep "${env_name}" >> "${env_path}";
    echo "export ${env_name}" >> "${env_path}";
done
 
chmod 600 "${env_path}"

To run this script at startup, I added this file to the i3 config file with the following command.

exec --no-startup-id "path to your script"

Then at the end of cron file, I added a new entry for the battery monitoring script.

To open cron file:

crontab -e

Now add the following line to the end of the file and save the file.

* * * * * bash "path to your script" >> "path to your log file"

Replace the path to your script (with double quotes) with your script path and the path to your log file with a path where you want to save your log file.

Now every minute, this script will be executed, and if your battery percent drops below the threshold value, you will be notified with a notification bubble.

I tested this procedure on Ubuntu 18.04 with i3. It should work on Arch Linux and other non-Debian distributions also, but the steps might be slightly different due to various reasons. Please comment if you face any issues with the setup.

Thank you for reading the article. Cheers 😄

Mistakes That I Made in NITC

Fri, 01 Jun 2018 04:51:45 GMT

Now that I am graduated from NIT Calicut, one more chapter of my life is over. It is time to move on to the next page, but before I begin that phase, I thought it would be nice, to sum up, the experience that I gathered in NIT Calicut. The last four years have been great for me. I learned a lot of new life lessons, but there were some decisions that I could have avoided. I take full responsibility for my actions and do not blame anybody for the same. Although I do not regret taking any of those decisions, I feel it is crucial to document all those lessons before they vanish from my otherwise average memory. This post is intended to serve as a document for my future self, but most of the content here should be able to help anyone who happens to read this post. Most of this content has been taken from my journal, which I maintained (not consistently) over the last two years.

Studies

From the first year itself, I had an impression from various online QA sites and some seniors that studies are not a big deal in college. That was a grave mistake which cost me a lot during my college life. I tried to compensate for it in the latter half of my graduation, but that was not enough.

While my friends used to learn by teaching each other, I never enjoyed studying in groups. I wish I had participated more in such group learning activities.

Most of the courses taught in traditional colleges like NITC are useless, in my opinion. However, Data Structures and Algorithms(DSA) was one subject which shows its importance again and again. It is one of those subjects where I should have put more attention. Although I was suffering from a fracture at the time of this course, some more efforts from my side could have changed things as they are today.

Wrong learning methods

I always tended to learn anything new using the tutorials available on the Internet. It is fine, but I never tried to do any projects/practicals to strengthen my learning. In my experience, it is always better to learn any programming language by doing projects in it.

My first programming language was C, which was taught in an introductory course in the first year. C is a relatively low-level language, and it helped me in understanding many concepts and working on many systems related stuff. Still, I feel that it is also one of the reasons that I am not able to think in higher-level abstraction. When I started learning Python, I was not able to think anything in abstract terms. I wish I had put more effort into learning Python or any other OOP language early in my college life.

Communication

I have always been an introvert person when it comes to communicating with people. I wish I had worked on improving this aspect of my personality. For the first two years, I never asked any seniors for advice, never asked any questions in the classroom, never discussed project ideas with any faculty. I tried to improve myself in the last two years, and I feel satisfied with my efforts. In my opinion, it is always better to ask a more experienced person for their advice than to take blind steps and regret later.

I was hesitant about talking to people from different linguistic backgrounds in my first and second years. Again thanks to my inability to communicate with new people. I improved this later, but first impressions are crucial, if not everything.

Friends and seniors

In my first year, I made a mistake by choosing to befriend some people who identified themselves as state seniors (people from the same north Indian state as I am). I was naive in identifying people at that time, but when I look back now, I feel that if I had avoided those people and invested that time in learning something related to CS, I could have gained more out of college. For a person who was entirely new for CS, that one year's time was very crucial, and I completely wasted it. It took me two years to rectify my mistake.

Not taking breaks

I never took breaks from learning new things. I always put my work ahead of my personal life. That was the wrong approach. However, I do not blame myself for this. I was new to such work-life equations, and I did not realize the requirement to maintain a proper work-life balance. I wanted to learn a lot of new things, and college provided me a suitable environment, but I feel that I exhausted myself many times. That resulted in my loss of interest from Computer Science also in the second year.

Writing and Reading

I was never a voracious reader in my childhood. I never tried to spend my free time with books. After coming to college, I saw many people reading books day and night. That inspired me to develop a new hobby of Reading. Although I still do not read a lot (just 5-6 books a year), it is better than reading none.

I started this blog in the second year. My intention was not to write. I just wanted to learn how to set up Wordpress and Jekyll. However, slowly I developed the habit of writing. I now consider it the biggest takeaway from college. Writing a blog has helped me learn new things and systematically explain things.

I tried to write a journal to document significant events in life, but I could not maintain it regularly. That is one thing which I would like to improve in the coming years.

Proxy Your Requests to the Backend Server With Grunt

Tue, 15 May 2018 11:44:13 GMT

This article was originally published on zeolearn.

If you are working on large projects, it is undoubtedly a good idea to have a build script or some task scripts to help to automate some of the repetitive parts of the development process. For JavaScript projects, Grunt serves a similar purpose. It is a JavaScript task/build runner that is written on top of NodeJS. Grunt can help you with automatically minifying your JavaScript or CSS files, or reload your browser on every file change. It can show you a comprehensive list of JavaScript errors, compile your SASS/LESS files into CSS files automatically, and many other things.

However, the most significant advantage of Grunt that I am going to discuss today is its ability to proxy your requests. For example, when you are developing your backend with anything other than JavaScript, you will face difficulty in accessing the backend data in your frontend without having to compile and deploy the code every time you make any changes. It is not possible with a typical web server setup because XHR requests are not allowed to be cross-domain by browsers due to Cross-origin resource sharing (CORS) limitations.

So, the problem here is as follows,

you are developing the UI of your applications using some frontend JavaScript framework (say Angular) with Grunt as the build runner, and the backend of your application is being designed in some backend framework other than JavaScript/NodeJS (say Laravel), you might face problems accessing the backend while running Grunt server.

It happens because the backend Laravel service runs on port 8000, and the front end development server runs on port 8080. The requests from the frontend server to the backend-server will result in same-origin policy errors due to the port difference. To fix this issue, we can set up CORS through a proxy on Grunt. This proxy will stand in front of your frontend server and the backend server and get the required data from the backend and pass it to your frontend while letting your browser think that you are all in the same domain.

Grunt has a module grunt-connect-proxy that exists to help to solve this issue. It delegates requests that match a given URL to the backend of your choice. So for example, you want to access your backend using the URL http://localhost:8080/api, you can write a proxy rule so that whenever your user tries to access this URL in a browser, the proxy will get the data from your backend and server it at this particular URL.

The procedure to set up the proxy is simple. First, you will have to add the proxy configuration to your Gruntfile.js. In this example, I am assuming that the backend server is running on the port 8000, and the Grunt server is running on the port 8080. This configuration will delegate all requests to http://localhost:8080/api to http://localhost:8000/backend.

connect: {
  server: {
    options: {
      port: 8080,
      base: 'public',
      hostname: 'localhost',
      livereload: true,
      middleware: function (connect, options, middlewares) {
        middlewares.unshift(require('grunt-connect-proxy/lib/utils').proxyRequest);
        return middlewares;
      }
    },
    proxies: [
      {
        context: '/api',
        host: 'localhost',
        port: 8000,
        https: false,
        rewrite: {
            '^/api': '/backend'
        }
      }
    ]
  }
}

Now register your Grunt server task to run the proxy on Grunt execution.

grunt.registerTask("server", function (target) {
  grunt.task.run(["configureProxies:server", "connect:server"])
})

Let me explain the above two scripts line by line. In the connect section of your Gruntfile, we add a new section called proxies. The options defined in the proxies section are explained below.

context: This is the context against which the incoming requests will be matched. Matching requests will be proxied to the backend server.
host: The host address where the backend server is running. The incoming requests will be proxied to this host.
port: The port where the backend server is running.
https: If your backend server is an https endpoint, then set this value to true.
rewrite: This option allows rewriting of URL when proxying. What this means is that when trying to proxy http://localhost:8080/api to the backend server, the URL will be rewritten as http://localhost:8000/backend. The object's key serves as the regex used in the replacement operation, and the object's value is the context of your backend server's service.

More options can be found in the documentation of grunt-connect-proxy.

You will also need to set up the proxy's middleware in the options section of the connect. The relevant code is as follows.

...
middleware: function (connect, options, middlewares) {
  middlewares.unshift(require('grunt-connect-proxy/lib/utils').proxyRequest);
  return middlewares;
}
...

Finally, include your proxy task in the server task. It is necessary to append the proxy task before the connect task. Also, make sure to specify the connection target in the configureProxies section. In our case, the connect target is server.

Now you can start your Grunt server via this configured proxy by typing Grunt server in the command line. You should see something like this in the console.

$ grunt server
...
Running "configureProxies:server" (configureProxies) task
Rewrite rule created for: [/^\/api/ -> /backend].
Proxy created for: /api to localhost:8000
 
Running "connect:server" (connect) task
Started connect web server on http://localhost:8080
...

The above output confirms that the proxy is working fine. Some of the example URLs are:

http://127.0.0.1:8080/api points to http://127.0.0.1:8000/backend
http://127.0.0.1:8080/api/x/y points to http://127.0.0.1:8000/backend/x/y

That's all. Now you will not face any problems getting data from any backend of your choice.

Writing Drozer Modules

Sun, 13 May 2018 11:40:13 GMT

This post is a result of my experimentation with Drozer. Drozer is a security testing framework for Android, developed by MWR Labs. According the Drozer's official documentation:

Drozer allows you to assume the role of an Android app and interact with other apps. It can do anything that an installed application can do, such as making use of Android's Inter-Process Communication (IPC) mechanism and interact with the underlying operating system.

Drozer modules are written in Python. The module performs operations on an Android device with the help of an agent app installed on the device. The agent app, by default, has permission to use the internet connection only. This permission is required so that the agent can open a ServerSocket on port 31415 (default). The agent will listen for the incoming connections on this port. The console will connect to the agent on this port.

Drozer modules are inherited Python classes. The parent class is defined in drozer.modules.Module. Drozer console provides commands to create a custom module repository, which is very useful for the local development of modules.

You can read more about the structure of a Drozer module here.

I will explain all the critical parts of a Drozer module with the help of a sample module. I will be implementing a module to record and save the sound from the inbuilt mic of an Android device.

I initialized a new module repository using the Drozer console using the following command.

dz> module repository create custom
Initialised repository at custom.

You will see a new directory custom in your current directory after executing above command. Navigate to this directory and create a new folder with any name. I prefer to name this folder same as my module name. In this folder, create a file __init__.py. Drozer identifies the folder as a module directory if __init__.py is present in the directory. Now you can implement your module in this directory.

To begin implementing our module, create a new file record.py in the module directory. Drozer has many different utility classes, which we can use to simplify our implementation. To use these utility classes (mixins), our module class must extend mixins using Python's multiple inheritance feature.

We first need to import all the required mixins. The mixins are stored in modules.common package in the Drozer source tree. After importing mixins and extending our class, the code will look like this. You can also import any other standard Python module here.

from drozer.modules import common, Module
import os, subprocess, time
 
class Record(Module, common.Shell, common.FileSystem, common.ClassLoader):

Now we will set up the metadata for our module. This information will help Drozer to organize and list our module correctly. We can define the name, description, author, date, license, path, permissions, and examples. Most of the available options are self-explanatory. But path and permissions require some explanation.

The path variable defined here is an array that contains the values for the namespace of the module. Drozer supports separate namespaces for each module. We can combine similar modules in the same namespace using this feature.

The permissions array variable contains all the permissions that this module will require for proper functioning. For example, our module will need permission to record audio on the device to work correctly. So we define this permission in the permissions array. The agent app on the device is required to have this permission. Otherwise, our module will throw an error.

The following snippet shows the metadata section of our module.

name = "Record sound from the inbuilt mic of an Android device."
description = "Record sound from the inbuilt mic of an Android device. The default save format is 3GPP. Relies on the agent having the RECORD_AUDIO permission."
examples = """
dz> run custom.record.record
Setting up recorder configuration...
Recording started
Press any key to stop recording
 
Recording stopped...downloading recording
Screenshot captured. Saved at location /home/yash/work/drozer/1524201166.3gp
"""
author: "Yash Agarwal"
date: "2018-04-14"
license = "BSD (3 clause)"
path = ["custom", "record"]
permissions = ["android.permission.RECORD_AUDIO", "com.mwr.dz.permissions.GET_CONTEXT"]

Now we can start implementing the heart of our module, the execute() function. This function will be invoked by Drozer when the module is run. Every action that the module is expected to perform should be implemented in this method.

The implementation of execute() method is slightly tricky and requires an understanding of different classes and methods provided by the Android API. As we are writing a module to record sound, we will look into the documentation of MediaRecorder class. Before reading further, go through the documentation about the use of reflection API in Drozer here.

The execute() function is given below.

def execute(self, arguments):
    # unique file names
    filename = str(int(time.time())) + ".3gp"
 
    # current working directory of Drozer console
    cwd = self.workingDir()
 
    # Magic of Reflection API !!!
    recorder = self.new("android.media.MediaRecorder")
    AudioSource = self.klass("android.media.MediaRecorder$AudioSource")
    OutputFormat = self.klass("android.media.MediaRecorder$OutputFormat")
    AudioEncoder = self.klass("android.media.MediaRecorder$AudioEncoder")
 
    recorder.setAudioSource(AudioSource.MIC)
    recorder.setOutputFormat(OutputFormat.THREE_GPP)
    recorder.setAudioEncoder(AudioEncoder.AMR_NB)
    recorder.setOutputFile("%s/recording.3gp" % cwd)
    recorder.prepare()
 
    self.stdout.write("Recording started\n")
    recorder.start()
    raw_input("Press any key to stop recording\n")
    recorder.stop()
    self.stdout.write("Recording stopped...\n")
    recorder.reset()
    recorder.release()
 
    # Download file from device to PC
    length = self.downloadFile("%s/recording.3gp" % cwd, filename)
 
    if length != None:
        self.stdout.write("Recording saved\n")
 
    else:
        self.stderr.write("Recording could not be fetched from the device.\n")

I followed the sample use case given on this page, to instantiate and use the MediaRecorder object.

After the recording is finished, we want to save this recorded media file to our computer. Drozer provides a method, downloadFile exactly for this purpose. This method returns the length of the data downloaded on success and None otherwise. We can use this information to test the success or failure of the fetching of the recording.

That's all. We have successfully implemented a Drozer module which can record the sound on an Android device without the knowledge of the user. Do you smell something fishy here? The whole idea here depends on that particular android.permission.RECORD_AUDIO permission that our agent app had. It allowed our module to record without user consent (actually, the user gave her consent unknowingly while installing agent app). Many apps nowadays ask for arbitrarily random permissions. Those permissions might not be related to the functionality of the app in any way, but because there is no method to install apps without granting these permissions, the users grant all permissions to these apps. That can be exploited very easily. This tutorial tried to show one of such exploitations.

Here are some exercises that you should try if you want to learn more about Drozer module development.

A module to initiate a call on a device.
A module to get the clipboard values on a device
Try finding a public exploit on Android forums such as XDA and implement that exploit as a Drozer module.

Slightly tougher one.

A module to terminate a call without user intervention (I do not know if it is possible to do this programmatically. If you implement this successfully, do let me know in the comments section.)

Thanks for reading. Cheers :)

So I Applied for GSoC

Wed, 28 Mar 2018 15:28:26 GMT

I first heard about Google Summer of Code (GSoC) in my second year (2016). It was quite fascinating to know that such a program exists (possibly because of that super attractive amount of money, $5500 at that time). I researched about it and sent emails to some mentors. The replies were mostly encouraging, but I still could not gather the courage to apply as I used to doubt my capabilities. Then last year (2017) again, I decided that I should try for GSoC one more time. I looked for the projects, started reading codebase of one project, but due to my lack of determination, I again did not apply.

In the summers of 2017, I got some time to think about it. I decided that for the next year, I will try everything to make sure that I apply for GSoC 2018. I started coding as much as possible and pushing my code to GitHub to build an impressive profile.

Now I can say that I am satisfied with my efforts to some extent. I did start some good open source projects. They are not very extraordinary, but something is better than nothing. I dived into Debian packaging also and volunteered for a FOSS event. Spreading awareness about FOSS is also contributing to FOSS. Isn't it?

Coming back to GSoC, this year, I have successfully submitted a proposal to the Debian Project. It does not matter whether I get it or not. I overcame one of my biggest fears of doubting my capabilities. It is something that I was trying to do for the last two years. This moment is worth celebrating.

Read about my GSoC project on Boot Many Machines via Bittorrent. It is related to something which I learned in SSL as SSL admin.

Read my project proposal on Debian Wiki.

Eagerly waiting for the results. 😄

Update(24/04/2018): Finally, I did not get into GSoC this time too. No worries. I anticipated the results and prepared backup plans. Looking forward to a productive summer. :)

FOSSMeet'18

Mon, 26 Mar 2018 06:27:14 GMT

I know I am very late in writing this post. It has been almost one and a half months since the last edition of FOSSMeet is over. I have been a part of FOSSMeet both as a participant and a volunteer. I want to share some of my observations, views, and suggestions through this post. I intend to keep a memory of my most favorite event of NIT Calicut. Juniors can read this post and think about ways to improve FOSSCell and FOSSMeet. I did the same by reading Kartik's post.

Before I start, if you have not read my post on the 2017 edition of the FOSSMeet, it is a good time to read that first. This post is going to be a follow up from where I stopped that post. Here is the link to that post.

I'll start by quoting some lines from the last paragraph of that article.

I hope that I will find time next year for FOSSMeet, although I would prefer to attend FOSSMeet as a participant observing everything silently rather than being a part of the organizing team.

These were my views after last year's FOSSMeet. I felt that I was not getting any benefit from volunteering for FOSSMeet. It is one thing that you help conducting an event, but on a personal level, I was not learning much. I wanted to attend talks and learn about free software philosophies, but because I was busy volunteering for workshops in SSL, I could not participate in any lecture. So I wanted to make sure that in my last FOSSMeet, I do attend all the talks and learn something new.

But fate had its plans for me. Few days after writing that post, Simsar, the coordinator of FOSSMeet '17, came to my room and asked me if I would like to become the next secretary of FOSSCell. Those who don't know, FOSSCell is the group of people in NITC, who are supposed to work on free and open-source contribution. But it is not very active for the last few years. Probably the seniors decided to give me a chance because I was quite active in providing my suggestions for the FOSSCell when I was in the third year and had shown my intentions to reinitiate FOSSCell. Anyway, I was not very enthusiastic about this opportunity and refused to accept it. I thought that the topic was closed and did not bother about it much.

Fast forward two months, I was at home thinking about the reasons that I could not get an internship were. I figured out some reasons (a topic reserved for a later post!) and decided that whatever happens, I will not commit these mistakes again. A few days later, Simsar messaged me on WhatsApp, asking whether I am still interested in the post of secretary. This time, I immediately accepted his offer. I was under the influence of my learnings from past mistakes. And that's how I became the unexpected (at least for my batchmates) secretary of FOSSCell.

I was placed in the first few days of placement season, so I was free on that front. CSEA had conducted a Debian packaging workshop at the very beginning of the semester, but I could not attend that workshop. So I messaged Pirate Praveen sir privately and asked him whether he can guide me about Debian packaging. As with most of the senior open source contributors, he was very enthusiastic about helping me. So that's how I restarted my journey in open source. It was a fun time learning about free software ideology from Pirate Praveen. We decided to conduct a small Debian Packaging hackathon in NITC. As it is a norm in CSE NITC nowadays about any such activity, people did not turn up at all for this hackathon. Only five people showed up for the hackathon. That was disappointing. Though, on a personal level, I was satisfied. I submitted two packages to the Debian unstable repository. We did not have any other activity for the remaining of that semester.

Okay, enough of the background!

My Observations

We opted for the HasGeek's Funnel for the proposal acceptance this time also. The funnel works quite well for our workflow. We did not get many proposals in the beginning. I had read somewhere on the web that FOSSMeet had a MiniDebConf once. That seemed like an excellent idea to the team. So I talked with Praveen sir, about the possibility of a separate Debian track during FOSSMeet. I thought that Debian packaging might be a good and easy way to introduce NITC students to Open-source software.

In previous editions of FOSSMeets, the organizers used to receive the feedback that the student audience is entirely novice for some sessions and the supposedly advanced session turned out to be a beginner session. We were well aware of this problem and thought of a way to solve this. The solution may be a controversial one because even I have mixed feelings about this solution. We decided to look at the GitHub profiles of each participant and then select a few participants rather than allowing each and everyone who is paying money to attend the FOSSMeet. Our experience from last year was awful in this aspect. Students from various colleges of Kerala came to FOSSMeet just to get the certificate. Such an audience was not contributing anything to FOSSMeet.

Now, let me explain why I think that this is not the right way to filter good participants. If I am not wrong, one motive behind organizing such student run conferences is to inspire beginners and college students to learn about free and open-source software and to allow them to learn about new technologies. Now, it should not matter if the person has any prior experience or not. In our case, we prioritized the previous experience part and neglected the will to learn. I know that it is tough to identify if a person is really interested in the event or just coming for the certificate, but still, there must be some better way to solve this issue. For now, this method is the best option available, and it worked well enough for our purpose.

The decision to allow only 150 participants was an excellent move. We had decided not to have parallel workshops and talks. This decision was a direct outcome of some of the bad feedback from the previous FOSSMeets. The decisions to check GitHub profiles and to restrict the number of participants made sure that this edition of FOSSMeet has the most appropriate (read qualified) audience for any session.

We publicized the event among CSED's first years. We even had a beginner level Git workshop for them. We were expecting that they will come to take part in the event. These events provide excellent opportunities for newcomers to learn about Computer Science and its various fields. I used to give priority to a CSEA event over any cultural event when I was in my first year. But, the turnout of first-year students for FOSSMeet was very less. I sometimes worry, how are these people going to continue organizing FOSSMeet. In the end, conducting this event will become a burden (of carrying a legacy) for them, and they will hold this event just like any other useless event that happens in NITC every weekend. We even made the entry, free for them; still, if they don't feel interested in the event, then they are at an apparent loss.

One other complaint that we received from the community was that the procedure of selecting proposals from the funnel is not transparent. In my opinion, it is true. The organizing team of FOSSMeet sit together and choose the topics based on the relevance of the topics and its relation to FOSS ideologies. Most of the proposals in the funnel were related to some technology or programming language. We aimed to give a chance to the talks which explain more about free software in general. Most of the proposals selected this year revolved around this theme with one or two exceptions. But I think it might be a good idea to publicly tell the factors that are taken into account while selecting proposals. FOSSMeet is an event about free software and the community, and it is the responsibility of the organizers to make sure that they adhere to the ideologies of free software, including transparency.

The decision to select philosophical topics might seem like a great thing. Still, in reality, the student audience is not very interested in listening to some random guy preaching about something which is not very relevant to them. Students want more technical knowledge, which they can get only from technical talks and workshops. I hope that future FOSSMeet organizers will be more careful about maintaining the right balance between technical and not technical discussions and workshops.

Some students from Amrita college complained about our selection of the same speakers every year. In my opinion, it is a very valid complaint. In my last three FOSSMeets, almost 3-5 speakers are giving talks/workshops every year. I am not saying that it is bad in any way, but the organizing team can start a bit early and invite some other prominent faces of the field from different parts of the country. The issue is that we begin contacting speakers very late and the people coming from far corners of the country or from abroad get very little time for making arrangements.

We used to have panel discussions in FOSSMeet. For the last two years, we stopped having one. I think that a panel discussion is a great way to learn about different viewpoints of experienced people. It will be good if the future organizers can squeeze one such panel discussion in the FOSSMeet schedule.

Now coming to my memories of this edition of the event, I liked the video conference by Bradley M Kuhn, the president of the Software Freedom Conservancy. The way he explained current issues in software freedom was fascinating. I was amazed by his humbleness. I was looking forward to the talk by RMS, but he could not deliver his speech due to some technical issues.

On a lighter note, It feels strange when you are working in a team where everyone except you speaks a language that you don't understand. I know that it is a very natural behavior, and I have no complains about it. But in my opinion, language is a tool of communication, and when you are not able to communicate your views to others, then there is some problem. When I think about these issues, I admire the forefathers of India for their farsightedness to make a foreign but universal language, one of the official languages of the Republic of India.

Conclusion

I feel at a loss because I got the chance to attend only three FOSSMeets (2015 edition never came to reality). It has been a great experience both as a participant and a volunteer for me over the last three years. I have written two posts about FOSSMeets and have tried to cover everything which I liked or disliked about this event. I might be very biased in the views expressed here as I was very disappointed with the lack of any activity from the FOSSCell side to introduce students about open source contributions. My lack of good communication skills made sure that the trend continues in my tenure also. I wish good luck to the next office bearers and hope that they will not continue this trend. 😄

Thanks to all fourth years – Sajmal, Navaneeth, Nithin, Kumar, Pavithra, Vrushabh, Nitin, and others for making the FOSSMeet a success. All the best to all juniors – Abhiram, Archana, Olive, Amruth, Gazala, Abhirami, Anupam, Madhumita, Adil, Vysakh, Arun, and others. Special mention to some second years also – Nirmal, Faris, Darshana, Kavitha, Naina, Vishnu, and others for helping us out in a smooth organization of the event.

Feeling relieved now, as this long overdue post is finally complete. If you find the article interesting, share your views in the comments section. Thanks for reading. 😌

Deactivated My Facebook Account

Fri, 23 Mar 2018 15:30:44 GMT

Today, I deactivated my Facebook account again. For the last few days, I was feeling that I am using Facebook excessively. It was affecting my work, my public life. I was using Facebook day in and day out. Today when I woke up in the morning, I first opened Facebook and didn't leave my bed for one hour. I realized that I had wasted one hour merely browsing through useless stuff. Immediately, I decided to deactivate the account for some time and analyze how much my life changes by one less distraction.

Coincidentally, it is the time of Ragam, the annual cultural festival of NIT Calicut. I introspect myself every year at the same time. For the last two years, this introspection is resulting in the deactivation of my Facebook account. It is the reason why I could finish all my pending projects in the summers.

I sometimes feel that after coming to NITC, I have become much more socially awkward and even more of a loner than I was before. I never enjoyed going into public events, but in college, that tendency seems to have increased a lot. Now, I do not feel comfortable with this state. I want to change this, but every time I try, there is some invisible force which pulls me back from expressing myself. NITC's environment was entirely different for a person like me, who always preferred to stay alone. So, I built a bubble around myself, where there is no one to disturb or doubt me. It was okay in the initial years of my campus life when circumstances were not in my favor, but now this bubble is an obstacle for me. I feel difficulty in coming out of this bubble.

There are only one and a half months left of my undergraduate life. I might not overcome this fear at this time. But I hope that the change of environment will help me cross this barrier. I have learned a lot of lessons from my mistakes in the past, and I hope that I will not repeat one of the biggest mistakes of my life.

I will write again about the changes I feel after deactivating Facebook. In the meantime, I will try to spend more time with my friends and less on my laptop. 😌

Develop a Theme for Hugo

Fri, 02 Mar 2018 05:57:05 GMT

This article was originally published on zeolearn.

Introduction

In this tutorial, I will show you how to create a basic Hugo theme. I assume that you are familiar with basic HTML, and how to write content in markdown. I will be explaining the working of Hugo and how it uses Go templating language and how you can use these templates to organize your content. As this post will be focusing mainly on Hugo's working, I will not be covering CSS here.

We will be starting with some necessary information about the terminology used in Hugo. Then we will create a Hugo site with a very basic template. Then we will add new templates and posts to our site as we delve further into Hugo. With very slight variations to what you will learn here, you will be able to create different types of real-world websites.

Now, a short tutorial about the flow of this post. The commands that start with $ are meant to be run in the terminal or command line. The output of the command will follow immediately. Comments will begin with #.

Some Terminology

Configuration File

Hugo uses a configuration file to identify common settings for your site. It is located in the root of your site. This file can be written in TOML, YAML or JSON formats. Hugo identifies this file using the extension.

By default, Hugo expects to find Markdown files in your content/ directory and template files in your themes/ directory. It will create HTML files in your public/ directory. You can change this by specifying alternate locations in the configuration file.

Content

The content files will contain the metadata and text about your posts. A content file can be divided into two sections, the top section being frontmatter and the next section is the markdown that will be converted to HTML by Hugo. The content files reside in /content directory.

Frontmatter

The frontmatter section contains information about your post. It can be written in JSON, TOML or YAML. Hugo identifies the type of frontmatter used with the help of identifying tokens(markers). TOML is surrounded by ---, YAML is by --- and JSON is enclosed in curly braces { and }. The information in the front matter of a content type will be parsed to be used in the template for that specific content type while converting to HTML.

I prefer to use YAML, so you might need to translate your configurations if you are using JSON or TOML.

This is an example of frontmatter written in YAML.

---
date: "2018-02-11T11:45:05+05:30"
title: "Basic Hugo Theming Tutorial."
description:
  "A primer about theme development for Hugo, a static site generator written in
  Golang."
categories:
  - Hugo
  - Customization
tags:
  - Theme
---

You can read more about different configuration options available for frontmatter here.

Markdown

The markdown section is where you will write your actual post. The content written here will automatically be converted to HTML by Hugo with the help of a Markdown engine.

Templates

In Hugo, templates govern the way; your content will be rendered to HTML. Each template provides a consistent layout when rendering the markdown content. The templates reside in the /layouts directory.

There are three types of templates: single, list and partial. Each kind of template take some content as input and transform it according to the way defined in the template.

Single Template

A single template is used to render a single page. The best example of this is about page.

List Template

A list template renders a group of related content. It can be all recent posts or all posts belonging to a particular category.

The homepage template is a specific type of list template. Hugo assumes that the homepage will serve as a bridge to all the other content on your website.

Partials

Partials are short code snippets that can be injected in any other template type. They are instrumental when you want to repeat some content on every page of your website. The header and footer content are good candidates to be included in separate partials. It is a good practice to use partials liberally in your Hugo site as it adheres to DRY principle.

Okay, Let's Start

So now that you have a basic understanding of Hugo, we will create a new site using Hugo. Hugo provides a command to generate new sites. We will use that command to scaffold our site. It will create a basic skeleton of your site and will give you a basic configuration file.

$ hugo new site ~/zeo
$ cd ~/zeo
$ ls -l
total 28
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:13 archetypes
-rw-r--r-- 1 yash hogwarts   82 Feb 11 11:13 config.toml
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:13 content
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:13 data
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:13 layouts
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:13 static
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:13 themes

Note: I will use YAML format for the config file. Hugo, By default, uses TOML format.

A small description of this directory structure:

archetypes: The archetypes contains predefined frontmatter format for your website's content types. It facilitates consistent metadata format across all the content of your site.
content: The content directory contains the markdown files that will be converted to HTML and served to the user.
data: From Hugo documentation

The data folder is where you can store additional data for Hugo to use when generating your site. Data files are not used to generate standalone pages; rather, they are meant to be supplemental to content files. This feature can extend the content in case your front matter fields grow out of control. Or perhaps you want to show a larger dataset in a template (see example below). In both cases, it is a good idea to outsource the data in their files.
layouts: The layouts folder stores all the templates files which form the presentation of the content files.
static: The static folder will contain all the static assets such as CSS, JS and image files.
themes: The themes folder is where we will be storing our theme.

We will edit the config.yaml file to edit some basic configuration of the site.

$ vim config.yaml
baseURL: /
title: "My First Blog"
defaultContentLanguage: en
languages:
  en:
    lang: en
    languageName: English
    weight: 1
MetaDataFormat: "yaml"

Now when you run your site, Hugo will show some errors. It is normal because our layouts and themes directories are still empty.

$ hugo --verbose
INFO 2018/02/11 11:20:59 Using config file: /home/yash/zeo/config.yaml
Building sites ... INFO 2018/02/11 11:20:59 syncing static files to /home/yash/zeo/public/
WARN 2018/02/11 11:20:59 No translation bundle found for default language "en"
WARN 2018/02/11 11:20:59 Translation func for language en not found, use default.
WARN 2018/02/11 11:20:59 i18n not initialized, check that you have language file (in i18n) that matches the site language or the default language.
WARN 2018/02/11 11:20:59 [en] Unable to locate layout for "taxonomyTerm":
...
...

This command will also create a new directory called public/. This is the directory where Hugo will save all the generated HTML files related to your site. It also stores all static data in this folder.

Let's have a look at the public folder.

$ ls -l public/
total 16
drwxr-xr-x 2 yash hogwarts 4096 Feb  11 11:22 categories
-rw-r--r-- 1 yash hogwarts  400 Feb  11 11:25 index.xml
-rw-r--r-- 1 yash hogwarts  383 Feb  11 11:25 sitemap.xml
drwxr-xr-x 2 yash hogwarts 4096 Feb  11 11:22 tags

Hugo generated some XML files, but there are no HTML files. It is because we have not created any content in our content directory yet.

At this point, you have a working site with you. All that is left is to add some content and a theme to your site.

Create a new theme

Hugo doesn't ship with a default theme. There are a lot of themes available on Hugo website. Hugo also ships with a command to create new themes.

In this tutorial, we will be creating a theme called zeo. As mentioned earlier, my aim is to show you how to use Hugo's features to fill out your HTML files from the markdown content, I will not be focusing on CSS. So the theme will be ugly but functional.

Let's create a basic skeleton of the theme. It will create the directory structure of the theme and place empty files for you to fill in.

# run it from the root of your site
$ hugo new theme zeo
$ ls -l themes/zeo/
total 20
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:30 archetypes
drwxr-xr-x 4 yash hogwarts 4096 Feb 11 11:30 layouts
-rw-r--r-- 1 yash hogwarts 1081 Feb 11 11:30 LICENSE.md
drwxr-xr-x 4 yash hogwarts 4096 Feb 11 11:30 static
-rw-r--r-- 1 yash hogwarts  432 Feb 11 11:30 theme.toml

Fill out LICENSE.md and theme.toml file if you plan to distribute your theme to outside world.

Now we will edit our config.yaml file to use this theme.

$ vim config.yaml
theme: "zeo"

Now that we have an empty theme, let's build the site.

$ hugo --verbose
INFO 2018/02/11 11:34:14 Using config file: /home/yash/zeo/config.yaml
Building sites ... INFO 2018/02/11 11:34:14 syncing static files to /home/yash/zeo/public/
WARN 2018/02/11 11:34:14 No translation bundle found for default language "en"
WARN 2018/02/11 11:34:14 Translation func for language en not found, use default.
WARN 2018/02/11 11:34:14 i18n not initialized, check that you have language file (in i18n) that matches the site language or the default language.
 
                   | EN
+------------------+----+
  Pages            |  3
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  0
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0
 
Total in 12 ms

These warnings are harmless in our case, as we are developing our site in English only.

Hugo does two things while generating your website. It transforms all the content files to HTML using the defined templates, and its copies static files into the site. Static files are not transformed by Hugo. They are copied exactly as they are.

The Cycle

The usual development cycle when developing themes for Hugo is:

Delete the /public folder
Run the built-in web server and open your site in the browser
Make changes to your theme files
View your changes in browser
Repeat step 3

It is necessary to delete the public directory because Hugo does not try to remove any outdated files from this folder. So the old data might interfere with your workflow.

It is also a good idea to track changes in your theme with the help of a version control software. I prefer Git for this. You can use others according to your preference.

Run your site in the browser

Hugo has a built-in web server which helps considerably while developing themes for Hugo. It also has a live reload and watch feature which watches for changes in your files and reloads the web page accordingly.

Run it with hugo server command.

Now open http://localhost:1313 in your browser. By default, Hugo will not show anything, because it cannot find any HTML file in the public directory.

The command to load web server with --watch option is:

$ hugo server --watch --verbose
...
...
                   | EN
+------------------+----+
  Pages            |  4
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  0
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0
 
Total in 11 ms
...
...

Update the Home page template

Hugo looks for following directories in theme's /layout folder to search for index.html page.

index.html
_default/list.html
_default/single.html

It is always desirable to update the most specific template related to the content type. It is not a hard and fast rule, but a good generalization to follow.

We will first make a static page to see if our index.html page is rendered correctly.

$ vim themes/zeo/layouts/index.html
<!DOCTYPE html>
<html>
  <body>
    <p>Hello World!</p>
  </body>
</html>

Build the site and verify the results. You should see Hello World! when you open http://localhost:1313.

Building a functional Home Page

Now we will create a home page which will reflect the content of our site every time we build it.

For that, we will first create some new posts. We will display these posts as a list on the home page and on their pages, too.

Hugo has a command for generating skeleton of posts, just like it did for sites and themes.

$ hugo --verbose new post/first.md
INFO 2018/02/11 11:40:58 Using config file: /home/yash/zeo/config.yaml
INFO 2018/02/11 11:40:58 attempting to create "post/first.md" of "post" of ext ".md"
INFO 2018/02/11 11:40:58 curpath: /home/yash/zeo/archetypes/default.md
...
...
/home/yash/zeo/content/post/first.md created

The new command uses an archetype to generate the frontmatter for new posts. When we created our site, hugo created a default archetype in the /archetype folder.

$ cat archetypes/default.md
---
title: "{{ replace .Name "-" " " | title }}"
date: {{ .Date }}
 
---

It is a good idea to create a default archetype in the themes folder also so that users can override the theme's archetype with their archetype whenever they want.

We will create a new archetype for our posts' frontmatter and delete the default archetype/default.md.

$ rm -rf archetype/default.md
$ vim themes/zeo/archetypes/post.md
---
title: "{{ replace .Name "-" " " | title }}"
date: {{ .Date }}
Description: ""
Tags: []
Categories: []
 
---

Create one more post in content/post directory.

$ hugo --verbose new post/second.md
INFO 2018/02/11 12:13:56 Using config file: /home/yash/zeo/config.yaml
INFO 2018/02/11 12:13:56 attempting to create "post/second.md" of "post" of ext ".md"
INFO 2018/02/11 12:13:56 curpath: /home/yash/zeo/themes/zeo/archetypes/post.md
...
...
/home/yash/zeo/content/post/second.md created

See the difference. Hugo used the theme's archetype for generating the frontmatter this time.

By default, Hugo does not generate posts with an empty content section. So you will need to add some content before you try to build the site.

Let's look at the content/post/first.md file, after adding content to it.

$ cat content/post/first.md
---
title: "First"
date: 2018-02-11T11:35:58+05:30
draft: true
Tags: ["first"]
Categories: ["Hugo"]
 
---
 
Hi there. My first Hugo post

Now that our posts are ready, we need to create templates to show them in a list on the home page and to show their content on separate pages for each post.

We will first edit the template for the home page that we created previously. We will then modify "single" templates which are used to generate output for a single content file. We also have "list" templates which are used to group similar type of content and render them as a list. The home page will show a list of last ten posts that we have created. Let's update its template to add this logic.

Update your home page to show your content

Now add your template code to themes/zeo/layouts/index.html.

$ vim themes/zeo/layouts/index.html $ cat !$ cat themes/zeo/layouts/index.html
<!DOCTYPE html>
<html>
  <body>
    {{ range first 10 .Data.Pages }}
    <h1>{{ .Title }}</h1>
    {{ end }}
  </body>
</html>

Hugo uses Go Template Engine. This engine scans the templates for commands that are enclosed between {{ and }}. In this template, the commands are range, first, .Data.Pages, .Title and end.

The template implies that we are going to get first 10 latest pages from our content folder and render their title as h1 heading.

range is an iterator function. Hugo treats every HTML file created as a page, so range will loop through all the pages created. Here we are instructing range to stop after first ten pages.

The end command signals the end of the range iterator. The engine loops back to the next iteration as soon as it encounters the end command. Everything between range and end will be evaluated for each iteration of the loop.

Build the website and see the changes. The homepage now shows our two posts. However, you cannot click on the posts and read their content. Let's change that too.

Linking your posts on Home Page

Let's add a link to the post's page from home page.

$ vim themes/zeo/layouts/index.html
<!DOCTYPE html>
<html>
  <body>
    {{ range first 10 .Data.Pages }}
    <h1>
      <a href="{{ .Permalink }}">{{ .Title }}</a>
    </h1>
    {{ end }}
  </body>
</html>

Build your site and see the result. The titles are now links, but when you click on them, it takes you to a page which says 404 page not found. That is expected because we have not created any template for the single pages where the content can be rendered. So Hugo could not find any template, and it did not output any HTML file. We will change that in a minute.

We want to render the posts, which are in content/post directory. That means that their section is post and their type is also post.

Hugo uses section and type information to identify the template file for each piece of content. It will first look for a template file which matches the section or type of the content. If it could not find it, then it will use _default/single.html file.

Since we do not have any other content type yet, we will just start by updating the _default/single.html file.

Remember that Hugo will use this file for every content type for which we have not created a template. However, for now, we will accept that cost as we do not have any other content type with us. We will refactor our templates to accommodate more content types, as we add more content.

Update the template file.

$ vim themes/zeo/layouts/_default/single.html
<!DOCTYPE html>
<html>
  <head>
    <title>{{ .Title }}</title>
  </head>
  <body>
    <h1>{{ .Title }}</h1>
    {{ .Content }}
  </body>
</html>

Build the site and verify the results. You will see that on clicking on first, you get the usual result, but clicking on second still produces the 404 page not found error. It is because Hugo does not generate pages with empty content. Remember I mentioned it earlier.

Now that we have our home page and posts page ready, we will build a page to list all the posts, not just the recent ten posts. This page will be accessible at http://localhost:1313/post. Currently, this page is blank because there is no template defined for it.

This page will show the listings of all the posts, so the type of this page will be a list. We will again use the default _default/list.html as we do not have any other content type with us.

Update the list file.

$ vim themes/zeo/layouts/_default/list.html
<!DOCTYPE html>
<html>
  <body>
    {{ range .Data.Pages }}
    <h1><a href="{{ .Permalink }}">{{ .Title }}</a></h1>
    {{ end }}
  </body>
</html>

Add "Date Published" to the posts

It is a standard practice to add the date on which the post was published on the blog. The front matter of our posts has a variable named date. We will use that variable to fetch the date. Our posts are using the default single template, so we will edit that file.

$ vim themes/zeo/layouts/_default/single.html
<!DOCTYPE html>
<html>
  <head>
    <title>{{ .Title }}</title>
  </head>
  <body>
    <h1>{{ .Title }}</h1>
    <h2>{{ .Date.Format "Sun, Feb 11, 2018" }}</h2>
    {{ .Content }}
  </body>
</html>

Adding top-level Pages

Okay, so now that we have our homepage, post-list page and post content pages in place, we will add a new about page at the top level of our blog, not at a sublevel like we did for posts.

Hugo uses the directory structure of the content directory to identify the structure of the blog. Let's verify that and create a new about page in the content directory.

$ vim content/about.md
---
title: "about"
description: "about this blog"
date: "2018-02-11"
---
### about me
 
Hi there, you just reached my blog.

Let's generate the site and view the results.

$ hugo --verbose
$ ls -l public/
total 36
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 12:43 about
drwxr-xr-x 3 yash hogwarts 4096 Feb 11 12:43 categories
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:20 css
-rw-r--r-- 1 yash hogwarts  187 Feb 11 12:43 index.html
-rw-r--r-- 1 yash hogwarts 1183 Feb 11 12:43 index.xml
drwxr-xr-x 2 yash hogwarts 4096 Feb 11 11:20 js
drwxr-xr-x 4 yash hogwarts 4096 Feb 11 12:43 post
-rw-r--r-- 1 yash hogwarts 1139 Feb 11 12:43 sitemap.xml
drwxr-xr-x 3 yash hogwarts 4096 Feb 11 12:43 tags

Notice that Hugo created a new directory about. This directory contains only one file index.html. The about page will be rendered from about/index.html.

If you look carefully, the about page is listed with the posts on the homepage. It is not desirable, so let's change that first.

$ vim themes/zeo/layouts/index.html
<!DOCTYPE html>
<html>
  <body>
    <h1>posts</h1>
    {{ range first 10 .Data.Pages }} {{ if eq .Type "post"}}
    <h2><a href="{{ .Permalink }}">{{ .Title }}</a></h2>
    {{ end }} {{ end }}
 
    <h1>pages</h1>
    {{ range .Data.Pages }} {{ if eq .Type "page" }}
    <h2><a href="{{ .Permalink }}">{{ .Title }}</a></h2>
    {{ end }} {{ end }}
  </body>
</html>

Now build the site and verify the results. The homepage now has two sections, one for posts and other for the pages. Click on the about page. You will see the page for about. Remember, I mentioned that Hugo would use the single template for each page, for which it cannot find a template file. There is still one issue. The about page shows the date also. We do not want to show the date on the about page.

There are a couple of ways to fix this. We can add an if-else statement to detect the type of the content and display date only if it is a post. However, let's use the feature provided by Hugo and create a new template type for the posts. Before we do that, let's learn to use one more template type which is partials.

Partials

In Hugo, partials are used to store the shared piece of code which repeats in more than one templates. Partials are kept in themes/zeo/layouts/partials directory. Partials can be used to override the themes presentation. End users can use them to change the default behavior of a theme. It is always a good idea to use partials as much as possible.

Header and Footer partials

Header and footer of most of the posts and pages will follow a similar pattern. So they form an excellent example to be defined as a partial.

$ vim themes/zeo/layouts/partials/header.html
<!DOCTYPE html>
<html>
  <head>
    <title>{{ .Title }}</title>
  </head>
  <body>
    $ vim themes/zeo/layouts/partials/footer.html
  </body>
</html>

We can call a partial by including this path in the template

{{ partial "header.html" . }}

Update the Homepage template

Let's update our homepage template to use these partials.

$ vim themes/zeo/layouts/index.html {{ partial "header.html" . }}
 
<h1>posts</h1>
{{ range first 10 .Data.Pages }} {{ if eq .Type "post"}}
<h2><a href="{{ .Permalink }}">{{ .Title }}</a></h2>
{{ end }} {{ end }}
 
<h1>pages</h1>
{{ range .Data.Pages }} {{ if or (eq .Type "page") (eq .Type "about") }}
<h2>
  <a href="{{ .Permalink }}"
    >{{ .Type }} - {{ .Title }} - {{ .RelPermalink }}</a
  >
</h2>
{{ end }} {{ end }} {{ partial "footer.html" . }}

Update the single template

$ vim themes/zeo/layouts/_default/single.html {{ partial "header.html" . }}
 
<h1>{{ .Title }}</h1>
<h2>{{ .Date.Format "Sun, Feb 11, 2018" }}</h2>
{{ .Content }} {{ partial "footer.html" . }}

Build the website and verify the results. The title on the posts and the about page should both reflect the value from the markdown file.

Fixing the date shown on About page

Remember, we had the issue that the date was showing on the about page also. We discussed one method to solve this issue. Now I will discuss a more hugoic way of solving this issue.

We will create a new section template to fix this issue.

$ mkdir themes/zeo/layouts/post $ vim themes/zeo/layouts/post/single.html {{
partial "header.html" . }}
 
<h1>{{ .Title }}</h1>
<h2>{{ .Date.Format "Mon, Jan 2, 2006" }}</h2>
{{ .Content }} {{ partial "footer.html" . }} $ vim
themes/zeo/layouts/_default/single.html
<!DOCTYPE html>
<html>
  <head>
    <title>{{ .Title }}</title>
  </head>
  <body>
    <h1>{{ .Title }}</h1>
    {{ .Content }}
  </body>
</html>

Note that we have changed the default single template and added that logic in post's single template.

Build the website and verify the results. The about page does not show the date now, but the posts page still show the date. We can also move the list template's logic to the index.html file of post section template.

Conclusion

We have learnt, how Hugo harnesses the powerful yet simple Go template engine to create the static site generator. We also learnt about partials and their excellent utilization by Hugo in the spirit of Don't Repeat Yourself principle. Now that you know how to make themes in Hugo, go ahead and start creating new beautiful themes. Best of luck for your endaevour.

My New Domain

Sun, 25 Feb 2018 04:40:04 GMT

I bought a new domain a few days back - yashagarwal.in. I was trying to buy this domain for the last two years, but it was already taken. My previous domain, yashagarwal.me that I got for free via GitHub education pack, was good enough for me, but the thought that I own a domain which is a top-level domain of some other country was in itself something which was not comfortable to me. So when I got the chance to buy this domain, I didn't delay. Hope this domain will remain my personal home on the web for years to come.

The domain is mapped with GitLab, and the SSL certificates are provided by Let's Encrypt. Let's Encrypt requires you to renew the SSL certificates every 90 days. That seems like a manual work to me. Moreover, my new registrar, GoDaddy, provides a complete API for their domain services. So I am planning to write a script that utilizes the APIs of GitLab and GoDaddy to deploy the SSL certificates automatically. I will probably use the DNS based authentication to verify the ownership of my domain, as that seems the only method that does not require any modification on the host side. I will write a post with all the details about the script, once I finish implementing it.

That's all for this post. See you next time. :)

Syncing Time on Windows & GNU/Linux Dual Boot Setups

Thu, 01 Feb 2018 07:24:34 GMT

This post is going to be one of those that I have written for my reference. Whatever I am going to mention in this post is not new. Everything has already been said and written many times on many websites and Linux forums.

So I will start by explaining the problem. When you try to dual boot your machine to run both GNU/Linux and Windows operating systems, you might have noticed that the time is not the same in both the operating systems. It is generally one operating system showing the correct time, and the other one showing the wrong time. It happens because Microsoft Windows thinks that the hardware clock (CMOS clock or BIOS clock) of the machine is using the local time (depends on your current time zone), and hence it doesn't do anything and shows you the same time. But most GNU/Linux operating systems (Ubuntu, Arch Linux, etc.) think that the hardware clock is set to track UTC. Hence the mismatch in the time happens. For example, assume that the current real-time is 10:22:51, and the hardware clock time is set to 10:22:51. Windows will interpret this time as local time and show 10:22:51, but Linux based systems will show 15:52:51 because they will understand this time as UTC. Of course, the above example is true if we assume time zone as India, which is +05:30 from UTC.

This issue can be fixed either from Windows or from GNU/Linux OS. I prefer to adjust the behavior of Windows to use UTC. It is much more convenient to use when traveling between different time zones. Please note that this method might not work or cause instability with older versions of Windows OS. I have tried this fix on Windows 10, and it works without any issues.

Open an Administrator Command Prompt by pressing ⊞ + x, then type a. This method of opening the Administrator Command Prompt does not work on Windows 7.

Now execute the following command:

reg add "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\TimeZoneInformation" /v RealTimeIsUniversal /d 1 /t REG_DWORD /f

Windows Time Service, which keeps the clock in Windows OS accurate, will still write the local time to the Real-time clock (RTC) regardless of the registry settings on shutdown. So I prefer to disable the Windows Time Service.

sc config w32time start= disabled

Now you may need to change the time in your BIOS to UTC time, although that depends on whether your Windows OS was showing the correct time before applying the above modifications. If yes, then changing BIOS time to UTC will make sure that both Windows and GNU/Linux convert hardware clock to local time.

References

Why a Sanskrit Shloka?

Tue, 30 Jan 2018 14:59:00 GMT

श्रूयतां धर्मसर्वस्वं श्रुत्वा चैवावधार्यताम्।
आत्मनः प्रतिकूलानि परेषां न समाचरेत्॥

If the entire Dharma can be said in a few words, then it is - that which is unfavorable to us, do not do that to others.

— Padmapuraana, shrushti 19/357-358

So when you landed on this blog, the first thing you might have noticed is that the home page of this blog has a Sanskrit shloka. Many people ask me, why did I choose to display a Sanskrit shloka on my blog.

There is no one answer to this question. There have been several incidents in my undergraduate life, which have given me a lot of good and evil memories. This shloka summarizes all those experiences that I earned in NIT Calicut during my undergraduate course. It reminds me, how I should treat others to receive similar treatment.

It was my first time away from home and family when I got admission in NIT Calicut. I was very naive about judging people. I got to know many new people in NITC. That is when I learned that not everyone is helping you. People are here to exploit you, and they will do anything to achieve their goals and to get success, whether it comes at the price of someone else's loss. It is the harsh truth of life, and the sooner one understands this, the better. I realized this in my second year, and that is when my life changed. Thankfully, I did not find other such people after my first year. However, life is very long, and I am bound to find such people at some point in my life again. I hope the experience that I earned in NITC will help me face that time.

Now, to the second part of the question, why Sanskrit? A quick Google search reveals that this shloka represents the concept of Golden Rule, which is common to most world religions, and Hinduism is no exception here. There is even a question about the relation of this shloka with Hinduism. I studied Sanskrit for six years till my 10^th standard and feel a connection to it. Besides, Hindi is my mother tongue, so Sanskrit was the obvious choice given that there is not much Hinduism related literature available in Hindi.

Edit(23/03/2018): I have removed the shloka from the front page now. I am keeping this post for historical purpose.

Arch Linux Installation Guide Part 2

Fri, 26 Jan 2018 10:15:27 GMT

In this post, I will continue from my last post and set up my newly installed Arch Linux for daily use. I am going to install some applications that I use on a day to day basis. Some of these applications are required for my current dotfile configuration setup to work properly. The choice of applications is highly opinionated and your preferences might be different.

If you had gone for installation via SSH option, then I would suggest you to edit your sshd_config file and disable root login. It can be a security risk otherwise.

Install a terminal based browser

Terminal-based browsers are very handy in cases when you are required to login into a captive portal and you don't have access to a graphical browser. We will install two different browsers, elinks and w3m.

sudo pacman -S elinks w3m

Install X server.

sudo pacman -S xorg

This will install minimal X desktop environment with fonts, in case, you want to test your system before installing any desktop environment.

Enable multilib repository for 32-bit package support

To enable multilib repository, uncomment the [multilib] section in /etc/pacman.conf.

[multilib]
Include = /etc/pacman.d/mirrorlist

Now upgrade your system.

sudo pacman -Syyu

Install video and touchpad drivers

sudo pacman -S xf86-video-intel xf86-input-synaptics

Install pacaur to fetch and install packages from AUR

sudo pacman -S expac yajl --noconfirm
cd /tmp
gpg --recv-keys --keyserver hkp://pgp.mit.edu:80 1EB2638FF56C0C53
curl -o PKGBUILD https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=cower
makepkg -i PKGBUILD --noconfirm
curl -o PKGBUILD https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=pacaur
makepkg -i PKGBUILD --noconfirm
cd

Install graphical browsers

pacaur -S firefox chromium

Install code editors

pacaur -S sublime-text-dev atom-editor-git visual-studio-code-bin neovim neovim-drop-in leafpad

Setup LAMP stack

Install Apache server

sudo pacman -S apache
 
# Make your user-directory available to apache server
mkdir ~/public_html
chmod o+x ~
chmod o+x ~/public_html
chmod -R o+r ~/public_html

# To enable virtualhosts, uncomment the following line in `/etc/httpd/conf/httpd.conf`
Include conf/extra/httpd-vhosts.conf

Add your virtualhost configuration in following file -

sudo vim /etc/httpd/conf/extra/httpd-vhosts.conf

To test the virtual hosts on you local machine, add the virtual names to your /etc/hosts file.

Install PHP

sudo pacman -S php php-apache

To use PHP with apache, open /etc/httpd/conf/httpd.conf and uncomment following line -

LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

and comment out the following line -

# LoadModule mpm_event_module modules/mod_mpm_event.so

Now add these lines to /etc/httpd/conf/httpd.conf:

# Add these at the end of `LoadModule` section.
LoadModule php7_module modules/libphp7.so
AddHandler php7-script .php
 
# Place this at the end of the `Include` section:
Include conf/extra/php7_module.conf

Install MySQL server

sudo pacman -S mariadb
 
# Initialize the MariaDB data directory prior to starting the service. To do so, run:
sudo mysql_install_db --user=mysql --basedir=/usr --datadir=/var/lib/mysql
 
# Then issue the commands to start the database server
sudo systemctl enable mariadb.service
sudo systemctl start mariadb.service
 
# To apply recommended security settings to your database, run
sudo mysql_secure_installation

Install PHPMyAdmin

sudo pacman -S phpmyadmin php-mcrypt

Enable mysqli, mcrypt, zip and bz2 extensions in /etc/php/php.ini.

Create the apache configuration file /etc/httpd/conf/extra/phpmyadmin.conf

Alias /phpmyadmin "/usr/share/webapps/phpMyAdmin"
<Directory "/usr/share/webapps/phpMyAdmin">
    DirectoryIndex index.php
    AllowOverride All
    Options FollowSymlinks
    Require all granted
</Directory>

Then include following in /etc/httpd/conf/httpd.conf

# phpMyAdmin configuration
Include conf/extra/phpmyadmin.conf

Now restart httpd service to apply settings.

sudo systemctl restart httpd

Once all these steps are done, your LAMP stack should be working.

Setup power management

Install tlp and some of its optional dependencies

sudo pacman -S tlp tlp-rdw bash-completion ethtool lsb-release smartmontools

Then enable tlp services

sudo systemctl enable tlp.service
sudo systemctl enable tlp-sleep.service
 
# mask some services for tlp to work properly
sudo systemctl mask systemd-rfkill.service
sudo systemctl mask systemd-rfkill.socket

Install i3 and other tools

All these tools are part of my i3 config with exception of the theme related packages. So installing them here will help me later while setting up the i3 window manager.

pacaur -S i3 rofi polybar xautolock powerline-fonts-git i3lock-fancy-git compton scrot feh dunst unclutter xfce4-power-manager numlockx lxappearance adapta-gtk-theme gtk-engine-murrine gnome-themes-standard termite

Fix Ugly Fonts 1

Fonts rendering is one area where Linux still lags behind Windows and OSX. It can be a nightmare for users to setup fonts properly in Linux. In Arch Linux, this is even worse. I found some tricks to improve the quality of font rendering on Arch Linux. Though this is far from perfect, it is manageable. Follow these steps on Reddit to fix font rendering. I use Noto Sans, Adobe Source Code Pro, and Microsoft fonts. My apologies, but I can't help here. Some websites still use Microsoft fonts.

Setup Python Environment

I use Python extensively and virtual environments are a must for my development setup. I use pipenv to manage my virtual environments. To install pipenv, you need to install virtualenv first. To install it, run the following command.

sudo pacman -S python-virtualenv

Now you are ready to install pipenv. Follow here to install the tool.

Install some other common tools

sudo pacman -S vlc openssh npm imagemagick git la-capitaine-icon-theme-git

Do not forget to setup npm to install packages globally without requiring sudo.

That's all! Your system should be in working condition now. Do check out my dotfiles if you want to set up your system like mine.

Hope you enjoyed the article. Cheers 😄

Footnotes

Make your Arch fonts beautiful easily! ↩

Arch Linux Installation Guide Part 1

Wed, 24 Jan 2018 16:19:05 GMT

Arch Linux is a Linux distribution known for its not-so-beginner-friendly command line installer, no ready-to-use system after installation and requirement of above average knowledge of command line. However, Arch Linux allows me to set up a system in my desired state in shortest possible time with least effort. This is why I keep coming back to Arch Linux even after some of its annoyances.

This guide is written primarily for my reference, as someone who has installed Arch Linux several times, I still can't remember all the installation steps perfectly. Most of the steps have been taken from Arch wiki and should work on other setups also.

All the commands are run in root shell unless otherwise specified.

0. Check your network connection

If you are behind a captive portal, use links to open browser and login into your network. For WiFi connections, use wifi-menu. LAN connections should not require any setup. The boot environment should automatically detect any wired connections. After connecting, test your connection by pinging any website:

ping -c 5 google.com

1. Setup SSH

This step is not mandatory, though I prefer to use this method to install Arch Linux, as it provides me the convenience of copying and pasting the commands directly from Arch wiki.

By default the Arch Linux root account password is empty. We need to set up a password for root account, which is needed for an SSH connection.

passwd

Now we need to change the setting to permit root login via SSH in /etc/ssh/sshd_config. Check that PermitRootLogin yes is uncommented in this file. If this line is not present there, add this to the end. Now start the sshd.service by issuing the command

sudo systemctl start sshd.service

Also, note the IP address of the target machine by inspecting the output of the following command.

ip addr

Pro tip: One liner to get only the IP address

ip -o -4 addr show | awk -F '[ /]+' '/global/ {print $4}'

Now on your host machine, connect to the target machine via SSH using the following command

ssh root@ip-address-of-target

2. Partition the disks

If Windows 8 or above is already installed on your machine, then your hard disk is probably using GPT partitioning scheme. In that case, use gdisk to partition your hard disk. ~~If you use fdisk on a GPT partitioned HDD, there is a possibility of data loss.~~ fdisk understands GPT partitioning scheme also.[1]

My preferred setup is to have one root partition and one home partition and use EFI partition created by Windows to install boot-loader. The root and home partition will be formatted using ext4 file-system and the EFI partition should be formatted using FAT32 file-system.

For this guide, I am assuming that the EFI partition is sda1, root partition is sda9 and home partition is sda10.

Now to format the partitions with ext4 file-system:

mkfs.ext4 /dev/sda9
mkfs.ext4 /dev/sda10

3. Mount the partitions

Now mount the root partition (sda9 in this case) to /mnt

mount /dev/sda9 /mnt

If you have created any other partitions in previous steps, mount them at appropriate locations.

mkdir /mnt/home
mount /dev/sda10 /mnt/home
 
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

4. Install the base file-system

To install the base system and some development tools, issue the following command.

pacstrap /mnt base base-devel

This will take a while to download and install. After it finishes, it will give you a bare-bone Arch Linux system with just the tools required to run a Linux distribution, no other software is installed.

5. Generate /etc/fstab

The /etc/fstab file stores the information about file systems of partitions and how to mount the partitions on system boot up. To generate this file, issue the following command:

genfstab -U /mnt >> /mnt/etc/fstab

If you prefer to use partition labels (sda1, sda9 etc.) instead of UUID, then use -L flag in place of -U.

6. chroot into the system

From the Arch wiki:

Chroot is an operation that changes the apparent root directory for the current running process and their children. A program that is run in such a modified environment cannot access files and commands outside that environmental directory tree. This modified environment is called a chroot jail.

At this step, we will go to the root of the newly installed system at /mnt and pretend as if we are logged into this system.

arch-chroot /mnt

7. Setup the time zone, locale, and hostname

Browse the /use/share/zoneinfo directory to find your location entries. My location is India, so I will use this command.

ln -sf /usr/share/zoneinfo/Asia/Kolkata /etc/localtime

To set the hardware clock:

hwclock --systohc

To set the locale for your system, open the /etc/locale.gen file and uncomment your language. or run the following command for the default en_US.UTF-8 UTF-8.

LANG=C perl -i -pe 's/#(en_US.UTF)/$1/' /etc/locale.gen

Now generate the localization with

locale-gen

Then set the LANG variable in /etc/locale.conf accordingly, or run the following command:

localectl set-locale LANG="en_US.UTF-8"

To set the hostname for your machine:

hostnamectl set-hostname your-host-name

To allow other machines to address the host by name, it is necessary to edit the /etc/hosts file to look like this:

127.0.0.1    localhost.localdomain          localhost
::1          localhost.localdomain          localhost
127.0.1.1    your-host-name.localdomain     your-host-name

8. Create user account

Before creating user account, set password for root account

passwd

Now create a local account for your user

useradd -m -G wheel -s /bin/bash your-user-name

This will set up your user account, create a home directory for your user, set the default shell to bash and add your user to wheel group, which is necessary to gain sudo access in later steps.

Set password for your user.

passwd your-user-name

9. Enable sudo access

This allows you to use root privileges without using the root account. To enable this, first open /etc/sudoers file

nano /etc/sudoers

Now uncomment the following line to enable root privilege for all the users inside wheel group:

# %wheel ALL=(ALL) ALL

Now you can safely disable root account

passwd -l root
 
# login into your user account
su your-user-name

From this point onwards, it is necessary to append sudo to any command that requires root privileges.

10. Install bootloader

My preferred bootloader of choice is grub. To install grub, we need to install following packages.

sudo pacman -S grub efibootmgr

Now install grub with the following command.

sudo grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=arch

Here --efi-directory is the folder where the EFI partition is mounted step 3 and --bootloader-id is the label that will appear in your UEFI boot menu entry.

This particular step is specific to my machine's hardware, you might not need to run this step. I need to add pci=nommconf to my kernel boot parameters in /etc/default/grub, otherwise tty prints error messages continuously.

Now run to generate grub configuration file.

sudo grub-mkconfig -o /boot/grub/grub.cfg

If you encounter any errors related to lvm during installation of grub, then follow these steps.

# come out of chroot
exit
mkdir /mnt/hostrun
mount --bind /run /mnt/hostrun
 
# back to chroot
arch-chroot /mnt
mkdir /run/lvm
mount --bind /hostrun/lvm /run/lvm

Now you can install grub without any errors.

11. Configure the network

By default, your current system cannot connect to the network in the current state. I prefer to use NetworkManager for my network management, even when I am not using GNOME. For wireless networking, install the following additional packages.

sudo pacman -S iw wpa_supplicant dialog networkmanager network-manager-applet dhclient

NetworkManager supports basic DHCP configuration. For full support, I have installed dhclient. NetworkManager also supports automatic wired connection detection and comes with curses based tool nmtui to setup wireless connection.

To enable NetworkManager to start at system startup

sudo systemctl enable NetworkManager.service

12. Reboot now

If you had performed the lvm troubleshooting steps during grub install, then

umount /run/lvm

Now exit from chroot by typing exit in the shell. Unmount all the mounted partitions with:

umount -R /mnt

Finally, reboot your machine by typing reboot and remove the installation USB drive. If you are not able to boot into your system at this point, boot from the installation media again and attempt to fix the installation.

If you can see a terminal with a prompt for your username, congratulations! You have completed the first step towards building your own system.

I will be writing about making your system usable and stable in the second part of this guide.

Hope you enjoyed the post. Stay tuned :)

My Own Configuration Manager

Sun, 21 Jan 2018 05:53:23 GMT

I have been using Linux since I was in my second year of undergraduate. My experiments with the dotfiles (configuration files) also started at the same time. For the uninformed, in Linux, it is common to configure a lot of settings and configurations within dotfiles. Dotfiles are files in a Linux user's home directory that begin with a dot or a full-stop character. This dot indicates to the operating system that these files are used to store the settings of programs like vim or shells like bash or fish to name a few.

In the beginning, I was keeping a manual backup of my dotfiles by copying them to a folder from time to time. But the list soon started getting huge, that it became complicated for me to keep track of the changes. Then I moved to symlinks. I started symlinking all the dotfiles from my folder to their usual locations. This setup worked perfectly fine, but as my collection of dotfiles grew, It became very cumbersome for me to symlink every dotfile manually.

I also tried a few tools built for this particular purpose. Some of them are vcsh, mr, and stow. These tools work just fine, but I was not willing to learn new tools just for maintaining my dotfiles. At last, I decided to write my tool to solve this problem. This way, there will not be any external dependency, and this tool will also become part of my dotfiles.

Design

The tool, in itself, is inspired by the UNIX tradition of keeping configuration files for the settings of the programs. This configuration system uses a JSON formatted dotfile.

Here is the source code for the configuration system. Let's have a look at the file structure of the repository.

|-- .backups
|   |-- 08-01-2018-15:47
|   |-- 08-01-2018-19:30
|   |-- ......
|-- configure.py
|-- current_status
|-- dotfiles
|   |-- dunst
|   |-- gtk-3.0
|   |-- i3
|   |-- ......
|-- dotfiles.json
|-- LICENSE
|-- README.md

During the initial setup, you need to edit the dotfiles.json file to suit your setup. A relevant section of the JSON file is given below.

{
  "pre": [
    {
      "name": "cloning repository",
      "command": "git",
      "subcommand": "clone",
      "argument": "https://github.com/yashhere/dotfiles.git"
    }
  ],
  "linking": [
    {
      "name": "bashrc",
      "src": "dotfiles/.bashrc",
      "dest": ".bashrc"
    },
    {
      "name": "bash_profile",
      "src": "dotfiles/.bash_profile",
      "dest": ".bash_profile"
    },
    {
      "name": "profile",
      "src": "dotfiles/.profile",
      "dest": ".profile"
    },
    {
      "name": "i3",
      "src": "dotfiles/i3",
      "dest": ".config/i3"
    }
  ]
}

As can be seen, the JSON file has an array variable linking, which can be used to set the paths for each configuration file and folder. The configure.py script also requires a dotfiles folder to be present in the current directory. The folder can be created manually, or if it is already version controlled on GitHub, then the script can clone it. For that, you can edit the pre section in the dotfiles.json.

Your dotfiles and config folders go inside the dotfiles folder. You need to copy all your current configurations to this folder to get started.

So, how does the script know where a file or folder will be linked? Simple, you need to edit the dotfiles.json file and add source and destination locations. For example, if you want to set up configurations of i3 to its original location (which is, $HOME/.config/i3), then you need to create a new JSON object in the linking array, like this.

{
  "name": "i3",
  "src": "dotfiles/i3",
  "dest": ".config/i3"
}

Here the name is used to identify the configuration file, the src parameter is the location of your config file/folder in the dotfiles directory, and the dest parameter is the final destination of the file/folder. Keen observers would notice that I have not used $HOME anywhere. It is understood that the configuration will go to the current user's home directory. So the dest is relative to the user's home directory, and src is relative to the directory from which the configure.py script is executed.

And you are done! Now, run configure.py, and all your dotfiles and folders will be symlinked to their correct place.

The current_status file saves all the symlink locations that are being managed by the script, for your easy reference and to debug any error.

Behind the Scenes

A lot to cool things happen behind the scenes. The script will check if any previous symlink exists at the given dest location. It removes any symlinks to avoid redundancy. If the dest already has any dotfile or folder, then it backs it up in the .backups under today's date and time before replacing it with a symlink to avoid any potential data loss.

I hope the article was useful. Cheers 😄

2017 - The Best Till Now

Sun, 31 Dec 2017 17:49:00 GMT

If I have to define 2017 in one word, that would be amazing. This year has been a life-changing year for me. I learned a lot of new things, had some excellent experience in the company of amazing people, had a great time in my academic life and SSL. 2017 has been a very significant year for me, and I want to record this memory by documenting some of the most amazing things that I learned and experienced this year.

Grades

Although I cared very little for grades for the most of my college life, this year changed my perception of grades. I finally understood that the grades are also equally important. I started 2017 with only one goal — getting good grades. I studied a lot in the first half of the year, and it is because of that time, I hold an excellent command over Computer Networks. If I have to choose any one semester for my placement, It would be 6th semester.

I invested most of my time for preparation of campus placements in the mid of the year. Reading Data Structures and Algorithms have proven to be very beneficial for me. However, the way I studied Data Structures, hasn't improved my skills in real-world problem-solving skills.

I did not do exceptionally well in academics in the late half of the year. Partly because of the after placement and "Dude, it is fourth year" type of environment around me. My final year project did not go in the right direction, and it is one of the very few disappointments for me in 2017.

Coding

It is one thing which I could never excel in my undergrad life. People say that coding is a skill and it requires practice. However, to continue doing that practice, one needs patience, that, unfortunately, I lack. This year continued to be the same as 2016. I could not overcome my fear of coding. I tried doing something different in the form of #100daysofcode, but couldn't continue it after 20 days because of my preparations for campus placements. I have again started writing code on a regular basis in late December. Looking forward to code more frequently in 2018.

Recently I started working on Debian Packaging. Praveen sir has always been very supportive of me and answered all my ludicrous questions very patiently. My conversation with him about free software philosophy has been very enlightening. Debian Packaging was a very satisfying experience to be able to give something back to the community. Now, I hold one package in the unstable Debian repository and two more packages in the pipeline for approval.

Placement

I joined NIT Calicut due to many reasons. Placements were undoubtedly one of the most important of them. Fortunately, I was among first few students to get placed in my batch. That moment came with a great relief in my life.

Reading

I had started this year with an aim to read ten books and keep track of my reading on Goodreads. In the end, I could not finish my goal. I did occasionally try to meet my goal, but my excessive reliance on soft copies of the books always distracted me. Now, it is not a surprise that I am unable to read anything on a screen. I got three hard copy books and finished two of them. However, I could not go past more than 20 pages of any digital book, that I bought on Kindle.

Writing

I have not written for the most of the year. I do think about writing occasionally and keep a note of all the ideas that I get. However, due to my busy schedule and my usual procrastination, I do not write that often. In 2018, I will try to write more often.

Health

It is one issue where I do not put enough stress. I can divide my year into two parts for this portion. Pre Birthday half and post Birthday half. I maintained a daily exercise routine for the first half of the year. That helped me in many ways. I felt more energetic, more productive and more resolute to troubles in life. I slept at ten each day and woke up early morning. It resulted in excellent health, which I had always lacked.

However, the second half of the year was disastrous. Due to weather and other unavoidable circumstances, I could not keep up my routine; once I came back to college. I started sleeping late again, and that resulted in me going to docter very often in the last two months.

Relations

I continued to feel bored in the company of people. It is somewhat strange that after getting into college, year by year, my interest in public events has decreased gradually. It may be because of the difference in my understanding of happiness and that of others.

I built good relations with some faculties in the department, my technical skills got me some admirers, who eventually became my friends. I made some new friends from MCA also this year. It has been a pleasant experience overall.

Looking forward to 2018 for, full of new experiences!

Setting Up SSH Agent in i3

Wed, 27 Dec 2017 14:51:09 GMT

In this post, I will write about the procedure to correctly setup SSH and GPG agents in the i3 window manager. To follow this post, you need to have ssh-keys and your private GPG keys ready. If you do not already have these keys with you, I will describe the process of creating the keys.

SSH

Generating an SSH key pair provides you with a public key and a private key. The private key should never be given to anyone and public key, well the name itself is self-explanatory.

To create a new key pair, open a terminal and paste the text below.

ssh-keygen -t rsa -b 4096 -C "your_email_address"

This command will create a new ssh key pair with the given email address as the label. Press Enter for any question asked. When it asks for the passphrase, type a strong passphrase, otherwise leave it blank to have no password.

GPG

You might need to download the GPG command line tools before following the below steps. Follow your distribution's documentation for more help.

Once you have downloaded the tools, open a terminal, and type the following command.

gpg --gen-key

You will see something like this. Enter 1 to select the default key choice.

gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
gpg: directory `/home/yash/.gnupg' created
gpg: new configuration file `/home/yash/.gnupg/gpg.conf' created
gpg: WARNING: options in `/home/yash/.gnupg/gpg.conf' are not yet active during this run
gpg: keyring `/home/yash/.gnupg/secring.gpg' created
gpg: keyring `/home/yash/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection?

At the next prompt, enter the key size. It is recommended to use the maximum key size of 4096 bits.

Enter the time duration for which the key should remain valid. Press Enter to specify the default selection, indicating that the key does not expire.

After verifying the information, enter your user information and a strong passphrase. Afterward, GPG will start generating your key. You will see:

We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse,
utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy.

You can now use the key (until it expires) to encrypt your data.

Setting up SSH Agent

Configuring i3

Update (26/05/2018): As pointed out by Saksham in the comments below, this step is not required for the setup of SSH in i3. This step can be safely ignored.

Open i3 configuration file and add an exec_always statement -

exec_always ~/.config/i3/scripts/gnome-keyring.sh

Obviously, you will need to change the path according to your OS. Now make a new file in ~/.config/i3/scripts with name gnome-keyring.sh and paste the below text in it.

eval $(/usr/bin/gnome-keyring-daemon --start --components=gpg,pkcs11,secrets,ssh)
export GNOME_KEYRING_CONTROL GNOME_KEYRING_PID GPG_AGENT_INFO SSH_AUTH_SOCK

(Assuming that you already have installed gnome-keyring)

Now, reload the i3.

Configuring SSH

Update (26/05/2018): This step is also optional. Thanks to Saksham for pointing it out.

Open ~/.ssh/config file and add following content to it -

Host *
  AddKeysToAgent yes
  IdentityFile /home/<your username>/.ssh/id_rsa

Replace <your username> accordingly.

Setting up .bashrc

I am not using a login shell, and I could not find any suitable method to source ~/.profile or ~/.bash_profile on login in i3. So I added my configuration to ~/.bashrc file. I know it is a hack, but it works well for me without much headache.

Open ~/.bashrc file and add following lines to the end of the file.

if [ -f ~/.ssh/agent.env ] ; then
    . ~/.ssh/agent.env > /dev/null
    if ! kill -0 $SSH_AGENT_PID > /dev/null 2>&1; then
        echo "Stale agent file found. Spawning a new agent. "
        eval `ssh-agent | tee ~/.ssh/agent.env`
        ssh-add
    fi
else
    echo "Starting ssh-agent"
    eval `ssh-agent | tee ~/.ssh/agent.env`
    ssh-add
fi

It will automatically start an ssh-agent if it is not already running. Otherwise, it attaches to a previously running agent.

Now log out and log in again to see if ssh-agent works. Open a terminal and run the command ssh-add -l. It will show you the hash value of your ssh-key, which is loaded by the ssh-agent.

That is all for today. Thank you for reading!

Setting up ALM Octane with Docker Compose

Tue, 26 Dec 2017 15:23:35 GMT

Recently, I got a chance to set up ALM Octane on one of my university servers for a course project. From the support page of ALM Octane:

ALM Octane is a web-based application lifecycle management platform that enables teams to collaborate easily, manage the product delivery pipeline, and visualize the impact of changes.

Precursor

My department insists on using open-source software (a plus point, indeed!). But ALM Octane has Oracle DB/MSSQL as a dependency. My professor was not very enthusiastic about installing a proprietary database on the server. So I came up with a different approach. I set up this database (free version of Oracle DB, the Oracle Database 11g Express Edition) in a Docker container. ALM Octane has three components – Oracle DB, Elastic Search, and the Octane Server itself. The problem was to handle the condition that these three components should be installed on different machines. I decided to use three separate Docker containers for this isolation and then configured them to communicate with each other with docker-compose.

Wait … Docker Compose?

(Skip this if already know about Docker and Compose)

Docker-compose is a tool to define and run multi-container Docker applications. Compose uses a compose file to configure the services used by the applications. Then all the services and the application can be run by using a single command.

So before reading this article any further, if you do not know about docker-compose, go and read about it.

The Problem

Three primary services are required for the proper functioning of the Octane Server – Octane, Oracle DB and Elastic Search. The difficulty was to set up adequate configuration options for these services and the appropriate setup for communication between them. I searched on Docker Hub for any pre-built images. Fortunately, I found some pre-built Docker images for my purpose. So I started writing my config file.

The Solution

Here is my config file for easy reference.

version: "2"
services:
  octane_oracle:
    image: alexeiled/docker-oracle-xe-11g
    shm_size: 2g
    mem_limit: 4g
  octane_es:
    image: elasticsearch:2.4
    environment:
      - ES_HEAP_SIZE=4G
    mem_limit: 4g
  octane:
    image: hpsoftware/almoctane
    ports:
      - "8080:8080"
    volumes:
      - /opt/octane/conf:/opt/octane/conf
      - /opt/octane/log:/opt/octane/log
      - /opt/octane/repo:/opt/octane/repo
    links:
      - octane_oracle
      - octane_es
    mem_limit: 4g
    env_file:
      - ./octane.env

The configuration options in the code are for a system with RAM of 8GB. The options must be tuned for the best performance before deploying on the production server.

In the code, I have exposed port 8080 of the Octane Docker container to the port 8080 of the host machine. It will make sure that we can access the Octane application on localhost:8080. The octane.env file contains three variables for easy site management.

SERVER_DOMAIN="your domain name"
ADMIN_PASSWORD="your password"
#This disables the minimum memory check to enable to run on smaller machines.
DISABLE_VALIDATOR_MEMORY=true

Now, if you try to run the container with docker-compose up, you will encounter various validation errors. It is because, before starting the server, you need to adjust some settings. If you noticed in the config file, there is a section to define volumes. These volumes are persistent and are used to keep changes saved between two docker-compose runs. I have mapped three volumes from the Docker container to the host machine. So you can now edit the files in the /opt/octane/ folder on the host machine, and changes will reflect in the Docker container. The file where you will have to make changes is /opt/octane/conf/setup.xml.

You can read the documentation of ALM Octane to find out which settings to be changed. Once you do the necessary changes, fire up the command docker-compose up and wait for some time. Docker Compose will finish processing, and the server can be accessed on http://localhost:8080 on the host machine. While deploying it on the production server, make sure that your network administrator has opened the port 8080 for your server. Otherwise, the site will not be accessible. Also, the docker-compose can be run in the background by issuing command docker-compose up -d.

Fixing Hindi Fonts in Arch Linux

Tue, 17 Oct 2017 16:25:53 GMT

When viewing Hindi content in any browser in Arch Linux, the rendering looks weird.

It doesn't look good, right! I'll try to fix this issue in this post. You might need to install the appropriate font support in Arch Linux before applying this fix. The suitable package for installing Indic Language support is ttf-indic-otf.

Now go to /usr/share/fonts/TTF and take the backup of two fonts FreeSans.ttf and FreeSerif.ttf. Now delete these two fonts from the directory. Restart the browser and see the difference.

This bug is reported in this bug report in 2011. I don't know why nobody has fixed it yet. Or maybe I was not able to find the proper solution. This workaround is also given in the same bug report.

I hope this helps. :)

Mounting NTFS partitions on Arch Linux

Tue, 14 Mar 2017 10:30:36 GMT

Yesterday I installed Arch Linux once again. A clean, bloat-free desktop with Budgie Desktop environment with some must-have open source tools. Everything worked fine except WiFi and some minor bugs in Budgie(I don't know whether it is a bug in Budgie or just a wrong setting). I also faced the problem of mounting Windows NTFS volumes on the user's wish. Arch Linux wiki has details about how to automount partitions on start-up. Still, I had a tough time to find out what exactly needs to be done to simulate the behavior of Ubuntu-like distribution on the mounting of NTFS drives. I got a hint from Arch Linux Wiki about the Polkit configuration setting, which can be used to allow a standard user to mount partitions. Here is a solution that I found after a long search on various Arch Linux Community pages.

You will need to install ntfs-3g, polkit and udisks2 to use this code. Please refer to Arch Wiki.

I think when using Gentoo, you will also need to compile the support for the NTFS file system in Kernel also. Please see here.

Add the following code to /etc/polkit-1/rules.d/10-udisks2.rules -

// Allow udisks2 to mount devices without authentication for users in the "wheel" group.
polkit.addRule(function(action, subject) {
    if ((action.id:= "org.freedesktop.udisks2.filesystem-mount-system" ||
        action.id:= "org.freedesktop.udisks2.filesystem-mount") &&
        subject.isInGroup("wheel")) {
            return polkit.Result.YES;
        }
    }
);
 
polkit.addRule(function(action, subject) {
   if ((action.id:= "org.freedesktop.udisks.filesystem-mount-system-internal") &&
        subject.isInGroup("wheel")) {
            return polkit.Result.YES;
        }
    }
);

Now you will be able to mount NTFS partition without any problem. :)

FOSSMeet'17

Mon, 13 Mar 2017 21:03:16 GMT

One more edition of FOSSMeet'17 was successfully organized in NIT Calicut recently. As an active member of the organizing team of this year's edition (though I sidelined myself at the end) and a keen but silent observer, I want to share my experience, ideas, and some observations through this post.

Marketing Website

We started planning the next edition of FOSSMeet sometime around September. Not many people were interested in planning. Anyway, Shrimadhav and I began working on the marketing website. The first design was straightaway rejected by Piyush and Simsar, which I feel, was fruitful. In the process, I learned some critical insights about planning, team management, and design of a user-friendly website. So I again started working on the site. The end product was clean, good looking, and simple (at least I was happy with it :P).

In the meanwhile, Amal and I started thinking about ideas, how can we restructure the FOSSCell, which had not seen a single activity (except those trademark events which were conducted just for formality) in years. I now understand that it was a mistake. It is a prerequisite about open source contribution that you should contribute to the products which you use in day to day life. When we started thinking about the FOSSCell, we didn't know about any of these points. We conducted a formal test(more of a filtering process), then we had a small meeting with the selected 2nd years about the FOSSCell, and we discussed the plan for next semester. Then everyone left for winter vacations.

A period of self-evaluation

In the winter holidays, we tried to have some IRC discussions, which again was somewhat successful because of the efforts from Shrimadhav and Simsar. It was a new experience for me also, but still, I tried to attend as much as I could. I also took one session on Git and Vim, which again was an entirely new experience for me.

In the winter vacation, I found some time to think about the purpose of FOSSMeet and FOSSCell and in what direction we were heading. I somehow understood that I was not qualified enough to guide someone about open source contribution or in general FOSS ideology, because I, myself was not contributing anything to open source community, and had no characteristic of a FOSS enthusiast. So I somehow lost my interest in FOSSCell and even in Linux. Shrimadhav asked me my ideas about FOSSCell activities, but I had no clue what to say. So that's how the FOSSCell again died without achieving anything significant. I feel it was my mistake. I still feel sad about this. :(

FOSSMeet time

So as the FOSSMeet's dates came near, people started coming for volunteering. It was good for the event, though. The funnel was already up last semester, and we were getting some good proposals also. FOSSMeet also went superbly. All the participants gave excellent reviews. There were some hiccups also, The Campus Internet Connectivity being the most significant one. There were some issues in SSL also, again, somewhere I hold an important responsibility. All in all, everybody praised the event and organization of the event. This year, we tried to reduce the use of plastics, and I feel that we were quite successful.

My Observations

There were some issues about the event, which I didn't like at a personal level. I sometimes think about what is the motive for conducting some activities in colleges. An event like FOSSMeet which attract the attention of all FOSS enthusiasts from all over Kerala has a huge potential of inspiring the young folks of NIT Calicut. We somehow succeed in that motive, but what about the situation after FOSSMeet. The enthusiasm about FOSS should not last just for two days, but we cannot blame others for the condition. I don't have a proper idea about the FOSS because there was no proper guidance from my senior batches. I learned many things in the workshops organized by CSEA, in fact, I first learned the proper use of Linux Shell in the Linux workshop conducted by CSEA (and FOSSCell, at least for formality). But you can't ask them to do everything. I think they are already overloaded. But there were no such workshops by FOSSCell.

Second, the curriculum of the NITC CSE department focuses more on the theoretical side of Computer Science, which is good. But it negatively affects student activities outside the classroom. That is one of the primary reasons why we cannot produce more GSoC participants or why don't we have more open source enthusiasts.

Third, one of the speakers in FOSSMeet mentioned one fundamental flaw in the structure of the organizing team of FOSSMeet. It doesn't have any representation from other branches. FOSSMeet has become an event for CSE folks, where people from different departments do not take any interest. I think publicity was a major issue here. I don't believe that there was any publicity done to explain FOSSMeet to the students of other branches. (If I am wrong here, please correct me).

Fourth, people are ready to volunteer for organizing an event, but they do not want to come and attend the workshops. FOSSMeet is held by individuals who have no previous experience of FOSS and have no intention to dive into open source community any time soon. FOSSMeet has become a tradition that is continued every year because it happened the previous year also. I do not feel that enthusiasm to promote FOSS in FOSSMeet because we are organizing it to keep the legacy. I am not saying that whatever effort organizers put in conducting FOSSMeet'17 was not significant. I am not the right person to blame them because I was also part of the same team. But still, I feel that FOSSMeet has lost its original motive to promote FOSS culture in NITC. It encourages more people from outside NITC, but inside NITC, it is just one of those many events organized by random clubs.

Conclusion

It's been an interesting one year for me after becoming SS Lab Admin because I started taking part in department activities that I always wanted to do. Being a member of the organizing team of FOSSMeet'17 was again a learning experience for me. I learned many new things, in SSL and outside SSL too. I committed some mistakes also in making the right decisions at the right moment, but that's how I learn. So no regrets. :) I hope that I will find time next year for FOSSMeet, although I would prefer to attend FOSSMeet as a participant observing everything silently rather than being a part of the organizing team.

Setting up Hugo automatic deployment to Github with Wercker

Wed, 22 Feb 2017 08:07:56 GMT

Recently, I again migrated my blog from Pelican to Hugo. So till now, I have experimented with Wordpress, Jekyll, Pelican, and Hugo. Without any doubt, Hugo is the simplest to set up. This time, I have setup Hugo in Windows, as I think, in my system, I reinstall Windows OS much less frequently than the Linux. So that way, it will be less painful for me to set up the blog again.

In this post, I will list all the process which I used to set up automatic deployment of Hugo generated site to Github pages using Wercker. In the beginning, I was trying to use Travis-CI, but then I read about Wercker somewhere. I was impressed with the integration of Wercker with Hugo and the availability of many its community-generated "steps" for the build and deploy process.

Hugo Docs already have a fantastic documentation for setting up Hugo with Wercker, but it is outdated. Other documentations available on-line is also obsolete. Wercker has changed many functionalities in its platform, which made it difficult for me to set up things correctly. But after hours of trial and error cycle, I was able to build and deploy my static files successfully.

Here I would like to share the issues I encountered and the tweaks I've made. The source codes of this site can be found here.

Project Pages or User Pages

Two types of sites are supported on Github Pages, User Site, and Project Site. User Sites will serve the files stored in the master branch of the repository https://github.com/user_name/user_name.github.io at the address https://user_name.github.io. For the Project sites, everything under the gh-pages will be served at the address https://repo_name.github.io.

My site is a User site, so I wanted all the static files to be saved in the master branch. As Hugo generates all the static files under public directory, I needed another branch to store my source files. So my made a new branch source, which will save all the source files for my blog. Don't forget to remove the .git folder from the theme folder. Otherwise, the build will fail at a later stage. You can try using the git submodule feature to avoid this issue. I created a repository yash2696.github.io in Github also.

git init                       #initialized git repository in site root
git checkout -b source         #created new branch source
git remote add origin [https://github.com/yash2696/yash2696.github.io](https://github.com/yashhere/yashhere.github.io)
git add .
git commit -m "Initial Commit"
git push origin source

Then I initialized my master branch as a orphan branch.

git checkout --orphan master
git rm -rf .
rm -f '.gitignore'
echo "#Your repository name" > README.md
git add README.md
git commit -a -m "Initial Commit"
git push origin master

Automatic deployment using Wrecker

It is straightforward to build a Hugo site. Invoke hugo command under your root directory, Hugo will create a public folder which will contain all your content, static files, etc. Then push this directory to Github, and voila, your site is up!

What if a single push to source branch can trigger all the process for you automatically. Here the magic of continuous integration(CI) comes into the picture. A free Wercker account can be easily created and hooked to the Github account and a new application from a chosen repository. After setting up everything, a push to the development branch will automatically trigger the Wercker. One of the most significant advantages of using Wercker is its extensive collection of user-made and well documented "steps". In this post, I will use two steps, build hugo and deploy to Github.

The first task is to create a wercker.yml file. It will tell Wercker which all actions, it should perform. Here is my wercker.yml for reference. In this, I have used two pipelines, build and deploy. Please follow the official docs for the more detailed steps. I will list all the problems which I face while setting up things properly.

Build

Following the official guide, I used this step to trigger Hugo to build HTML pages. I had already removed git repository information from the theme folder, so this step finished successfully. If you haven't, you may add the following piece of code in your build step.

- script:
    name: install git
    code: |
        apt-get update
        apt-get install git -y
    - script:
        name: initialize git submodules
        code: |
            git submodule update --init --recursive

Deploy

There is no concept named "Add Deploy Target" in Wercker as of now. Most of the on-line tutorials follow this process which is outdated. Now Wercker uses a concept called "Workflows in Pipelines".

For new interface, even if you add a deploy stage in the wercker.yml, you will have to create a new pipeline deploy under the Workflows tab. After creating the pipeline, the "YML Pipeline name" must be set to the deploy stage name, which in this case is deploy.

In deploy stage, I used this step(broken link) to deploy the built site to Github. Each pipeline starts from scratch, so for the deploy pipeline, the git package needs to be installed again. One also has to set up the environment variable $GIT_TOKEN to each pipeline, acquired from Github setting.

You need to generate a new access token for your deploy stage from Github settings.

After adding the deploy stage, add the token you obtained from the Github to Environmental Variables in deploy pipeline.

On the next push to your development branch, Wercker will automatically build the site and deploy it on Github Pages.

Setting up Python Development Environments

Sun, 09 Oct 2016 00:00:00 GMT

Recently I was searching for Python projects on Github for contribution. Every single project I found, had a thing common among them. In every project's contribution guide, it was asked to set up the virtual environment for the project. What the heck is this virtual environment and how does it work?

As a beginner to open source projects, the problem I faced, in the beginning, was how to set up the development environments for the projects I was looking at. I searched the Internet, I found some articles, but they were not complete. So I decided to write this guide, which will be useful for me in future also.

Python uses pip for package management.

Installing pip

pip depends on setuptools library, which is in official Ubuntu repositories. To install it for python2 -

sudo apt-get install python-setuptools

Then install pip using -

sudo apt-get install python-pip

and for python3 -

sudo apt-get install python3-setuptools

Then install pip using -

sudo apt-get install python3-pip

It should install pip on your system for both python versions. pip is very easy to use. It will take care of every single package you may require for your project.

Installing a package using pip

# it will search and install [package]
pip install [package]
pip install django

If you are using python3, then don't forget to use pip3.

pip can be used to install a specific version of package also.

# it will search and install [package] with [version]
pip install [package]==[version]
pip install django==1.6.5

Uninstalling a package using pip

# it will search and uninstall [package]
pip uninstall [package]
pip uninstall django

upgrading a package using pip

# it will upgrade [package] to latest version
pip install --upgrade [package]
pip install --upgrade django

Creating list of all packages with pip

It is one of most used and most useful feature of pip. It allows you to make a list of all the dependencies of your project.

# it will output the file to current directory
pip freeze > [file_name.txt]

All these commands above will install the packages globally. But that's not what is desired. virtualenv comes to our rescue here.

Virtualenv

virtualenv solves a very particular problem; it allows multiple python projects that have different and often conflicting dependencies, to coexist on the same system.

virtualenv solves this problem by creating different isolated development environments for your projects. An environment is a folder which contains everything; your project needs to work properly.

Installing virtualenv

By default, if you install virtualenv using pip, it will use system's default python to create virtual environments. To overcome this problem, we will install virtualenv using ubuntu package manager.

sudo apt-get install python-virtualenv

Installing virtualenvwrapper

virtualenvwrapper provides some set of commands which makes working with virtual environments much easier.

To install it -

sudo pip install virtualenvwrapper

pip, virtualenv and virtualenvwrapper are the only packages which you will need to install globally. All other per project packages will be installed in respective virtual environments.

virtualenvwrapper also places all your virtual environments in one place. It makes working with projects very easy.

Now open your .bashrc and add these two lines to the end -

# All your projects will be saved in python-dev folder
export PROJECT_HOME=~/python-dev
 
# ~/python-dev/virtualenvs will contains python interpreters for each project.
export WORKON_HOME=~/python-dev/virtualenvs
 
# source the virtualenvwrapper script
source /usr/local/bin/virtualenvwrapper.sh

You can change python-dev to any name you wish. Your virtual environments will be created at that location.

Now restart your terminal to source the .bashrc or use -

source .bashrc

Basic Usage

Create a virtual environment -

mkvirtualenv myproject

It will create myproject folder in the python-dev directory. To activate this project -

workon myproject

Alternatively you can create project using mkproject command. It will create a virtual environment as well as a project directory in the $PROJECT_HOME, which is cd-ed into when you workon myproject.

Don't forget to deactivate current project when you switch between different projects.

To deactivate a project -

deactivate

To delete a virtual environment -

rmvirtualenv myproject

List all environments -

lsvirtualenv

it will also list all virtual environments -

workon

Please refer to virtualenvwrapper documentation for full list of virtualenvwrapper commands.

virtualenvwrapper also provides the tab-completion feature which is very handy when you have a lot of projects to work with.

That's it. Hope you liked the post. 😄

Custom Arch Linux setup with Openbox

Thu, 09 Jun 2016 10:20:36 GMT

After my summer vacation started, I bought a new laptop, and the first thing I did was to install Arch Linux on it. After a standard arch installation procedure, I started putting together my desktop environment, beginning with ArchLinux and Openbox, and then piecing all pieces together to build a proper desktop environment. Building a desktop this way follows the Unix Methodology; have software that each does one thing well, and when you put them together, you get something amazing.

When I first installed Arch, I had several choices. I tried xfce and gnome as my desktop environments. While both of them have their pros and cons, I ended up liking none. The sole reason was the dependencies; these two software bring with them. My only intention of installing Arch was to have something, which I control, not like Ubuntu, where you are forced to use the preinstalled software and when you try to uninstall something, you fear of breaking some other program.

So I end up going the route of building something up from scratch using Openbox as a base. At this point, I have a fast, lightweight desktop that is exceptionally stable and is genuinely my desktop as I have built it from the ground up, choosing every application.

Window Manager

After the installation of Arch, the first thing I did was to install openbox, a tiling window manager. You may also try other window managers like awesome, i3, and fluxbox, etc. but I chose Openbox because it is used everywhere. Thus there is a plethora of information out there about customizing it.

For our window manager to show up, it needs to be added either in a login manager or user's ~/.xinitrx file. There is a default version of this file that contain some code which we may want to retain, so rather than making this file from scratch, copy the default version of the file to the user's home directory.

cp /etc/X11/xinit/xinitrc ~/.xinitrc

Now add the command exec Openbox-session to the end of the file. Remember that the system will ignore the lines following this command. An Openbox session should be able to be started by entering the command startx.

An openbox session can be started automatically upon login using shell's startup script. Add following to the shell's startup script, ~/.bash_profile for bash and ~/.zprofile for zsh.

[[ -z $DISPLAY && $XDG_VTNR -eq 1 ]] && exec startx

Configuring Openbox

Setting up openbox is quite simple. Four files make the basis of openbox configuration. They are rc.xml, autostart, menu.xml and environment. These files controls everything about the window manager.

rc.xml: Determine the behavior and settings of overall Openbox session
autostart: Contains a list of applications to be launched with the window manager
menu.xml: It makes the right-click context menu of the desktop
environment: Can be used to export and set relevant environmental variables

For the configuration of Openbox, it is necessary to create a local Openbox profile in the user's home directory. A global configuration file can be found in /etc/xdg/openbox, which is well documented, and copying the global configuration to the user's directory will give a good starting point to start customizing.

cp -R /etc/xdg/openbox ~/.config/

These files can be edited by hand, although some graphical tools are also available, use of these may be desired.

Themes and Appearance

obconf and lxappearance-obconf can be used to configure appearance and theme of openbox session. There are quite a few themes available in openbox-themes package. My personal favorite is Numix-themes and Numix-icon-theme-git (AUR).

To see changes after editing a configuration file, the Openbox needs to be refreshed. It can be done with the reconfigure command.

openbox --reconfigure

Menus

The type and behavior of Openbox menus, accessible by right-clicking the background, can be changed using ~/.config/openbox/menu.xml file. Openbox provides two kinds of menus, Static, and Dynamic menus (Piped and Generators)

Static menus are hardcoded in XML and is stored in the menu.xml file. Whenever you install a new application, you will have to update the XML file to update the menu manually. It is a viable solution if the apps are not installed on a day to day basis.

Pipe menus are the sections of the Openbox menu that Openbox creates on the fly by running a generic script and using its plain text output as menu entries. This scheme can be used in different ways, like adding a mail checker in the menu or adding a weather forecast menu. You can check openbox pipe menu page for more information.

Generators are the most convenient type of menus. These can be found in most desktop environments where applications show up in the menu automatically. If applications are being installed regularly, then this will probably be the preferred choice.

Static Menus

The process of making these menus can be automated by static menu generator like obmenu. It will generate static menus from installed applications by looking into certain directories. Others available tools are menumaker, obmenu and xdg-menu.

Dynamic Menus

Dynamic menus give the same kind of functionality most people are used to. So it was my preferred choice. They can be used to generate full, complex menus on the fly. One of the most popular application for generating dynamic menus is obmenu-generator. Though it is not officially connected to Openbox, it is widely used.

In order to have obmenu-generator make a menu on demand, the menu.xml file should contain the following code as the only entry.

<?xml version="1.0" encoding="UTF-8"?>
<openbox_menu>
    <menu id="root-menu" label="OpenBox 3" execute="/usr/bin/obmenu-generator"></menu>
</openbox_menu>

Panels and Taskbars

You can get any panel like xfce4-panel or the simple and customizable tint2. I chose tint2 because it very closely follows the openbox spirit of having easy customization with plain configuration files. tint2 package also containg a graphical tool tint2conf for customizing panel and you can obviously go right to the tint2 configuration file and edit it there. This also means it is easy to pick up your configuration file and move it to another computer, or restore an old configuration since it is as simple as pasting a text document in the right place. tint2 can also have sections for each individual desktop and has various small widget like programs that can be used with it such as a calendar, gsimplecal, a volume application, volumeicon, a battery indicator, cbatticon, a network indicator like wicd which also have a system tray icon and a mini task tray. To start tint2 with openbox, you need to add tint2 & to the autostart file.

File Manager

Some file managers deeply integrate into desktop environments, and when installed with Openbox, they end up pulling multiple dependencies, which is certainly not desired. Our aim behind building an Openbox set up was to have minimum dependencies so that we can get a lightweight desktop. However, there are several file managers available that fit into the Openbox very well. In my choice, pcmanfm is the best choice here. pcmanfm is a lightweight and fast file manager with network and thrash support. pcmanfm can also manage desktop icons and wallpaper setting, although if you plan to use conky in your system, then you may not want pcmanfm to manage desktop for you. The reason is that pcmanfm treats desktop as a window, so when you switch to desktop, the conky hides behind the desktop window. To get the icons and wallpaper, you can use other standalone tools such as idesk and feh or nitrogen. To start pcmanfm is the desktop mode, add the following to the autostart file.

pcmanfm --desktop &

and to stop it for managing desktop

pcmanfm --desktop-off &

Many functions present in modern file managers require the installation of a few additional programs. gvfs gives you the ability to mount volumes and have trash control. It is not mandatory to have, but it allows on-demand mounting completely from within the file manager, as opposed to having to drop down to the command line. A disk-volume manager such as udisks2 will also be wanted, and for auto-mounting removable disks, udiskie works well with Openbox. udiskie can be added to Openbox's autostart file like any other application.

Composition

Openbox does not provide native support for composition, although having a compositor may prevent screen flickering and other distortion in applications like oblogout. Some options work great with Openbox. The one which I am using and quite happy with is compton. It can be used with or without any initial configuration, although there are a lot of different configuration options if you decide to customize the setup. To start compton in background with shadows, use this in autostart file.

compton -b -c &

Wrapping up

While setting up my desktop using Openbox ended up being much more work than I could have with typical desktop environments. But now I have a much more stable desktop, about which I know, what is running in my system, and that my system is not cluttered with any software that I never use.

While it might not be for everybody having to configure every little bit of the desktop, I genuinely feel I have ended up with excellent user experience, and a system that is customized precisely to fit my needs.

References

Arch Linux Wiki

A good Sublime Text setup

Thu, 14 Apr 2016 10:55:35 GMT

So, after a hectic day, good news finally came. I have been selected as a Lab Administrator for the Software Systems Lab of my college. Cheers!!!

Let's come to our today's topic on configuring Sublime Text Settings.

There is just one rule you must follow while designing your own editor preference configuration.

Don't put any lines in your configuration that you don't understand.

You will find tons of online tutorials that contains all kinds of awesome hacks to make your sublime text experience better but the worst way to make your development environment better is just to borrow the configuration from someone else.

Spending your time in actually understand what is happening behind the scenes in the construction of your editor is immensely invaluable. It is similar to the increased information retention that you experience when you copy something from the board.

So first, take some background of what we are going to do today. I am using Sublime Text 3 – dev version, but most of the instructions are similar for Sublime Text 2 also.

Okay, first open the sublime text, then go to Preference → Settings → User.

So this is your configuration file where you can put all your custom preferences. There are other configuration files also, which can be found in Preferences.

You can also find all settings for reference in Settings → Default file.

Here is my Settings → User file. Feel free to take inspiration from it, and make your sublime text experience unmatchable. I have commented every setting which is self-understandable.

https://gist.github.com/yashhere/dea6728f88865666c20ff1c35e90bd0e

Wrapping It Up

I still stand by my platitude that

Don't put anything in your configuration file you don't understand!

That's all for today. Thanks for reading.

Searching the goal

Sun, 10 Apr 2016 00:33:16 GMT

A boy, probably of age 16, just cleared his 10th board exam. He saw people working on computers, and that fascinated him. So he had only one ambition. You got it, right? One fine day, suddenly his friend's mother asked, "So what about the future? You are going to enter in class 11th. It is the right time to decide about your future. What do you want to become, a doctor or an engineer?" The boy had no clue how big that question was. What the cost would have been. He simply nodded and said, "I like computers very much so I would go for engineering."

That's it. He decided his fate.

So here comes the day

An accident

After that day, he got into a coaching institute which promised a good rank in JEE. He was told that he would have to study consistently and regularly. The boy, being a determined one, started studying day and night.

This continued for two years. Finally, after giving 12th board exams, he gave JEE. Unfortunately, he didn't even get into extended merit list.

Devastated, he pledged not to give up. Oh, that stupid silly guy! He decided to try for one more year.

He joined another institute in his city. Again the same routine, studying day and night to fulfill his dreams.

Result: Same, Fail!

But somehow he got a decent rank in JEE Mains thanks to his excellent performance in board exams.

A disaster

Now he had to face one more question. Which college and which branch? For him, the answer was simple. Go to any South Indian college thanks to the influence of his cousins on him (who were very successful in their lives, of course!). He chose Computer Science and Engineering because

He loved computers.
Everybody said, "Bahut scope hai is branch me."
Every top ranker of this country choose this branch.

So the decision was clear. He packed his bag and left home for a NIT situated some 2200 kms far, the comp-sci branch.

The coding disaster

He reached the college. He loved new people, new place, new environment, everything was new, right!. Studies started. He got good grades initially until the real coding and Computer Science related stuff started as courses. His grades fell drastically, his morale too. He got good marks enough to pass the courses. He tried giving his best, but all concepts seemed so alien to him as they wouldn't stay inside his head.

That guy is now about to complete his 2nd year and just thinking what went wrong with him. He is still trying to keep his journey on the right track. Whether he will be able to finish it successfully or not, only time will tell.

At last one more year is coming, and it's story will also come here next year.

Inspired by this Quora Post

Wanna get insulted by sudo

Sun, 03 Apr 2016 12:20:36 GMT

You might have tried many Linux easters eggs for fun, but you are going to love this hack which makes sudo insult you.

Confused what I am talking about? Here, take a look at this gif to get an idea of how sudo can insult you for typing in the incorrect password.

Now you might be thinking, why would anyone want to take insult? Afterall, nobody likes being insulted. For me, it is just another way to have fun with Linux, and anyway, this is way better than the plain "You entered a wrong password" error message. So let's learn how to do this.

Enable insults in sudo

You can enable the insults feature in sudo by modifying the sudo configuration file. To open the sudo configuration file, launch a terminal and type the following command.

sudo visudo

It will open /etc/sudoers configuration file in the terminal, in vim text editor if you have configured it as your default editor. In distros like Ubuntu, it will be opened in nano. Now you will have to find the section where the defaults are listed. Most probably you will find it at the top. Now find the line that starts with Defaults and append the word insults to the end of the line (any addition to the line is comma separated). If this line is not present then add the following line to the section

Defaults insults

(Always use visudo as it has a self-check system which will save you from messing up things)

Now save the file. If you are using vim, then use Ctrl+X to save the file and quit the editor and if you are using nano then use Ctrl+X to leave the editor. At the time of quitting, it will ask you if you want to save the changes. To keep the changes, press Y.

Once you have saved the file, go to terminal and type the following command to clear the old password from sudo's cache.

sudo -k

That's all. Use any command with sudo. Deliberately type a wrong password and enjoy abusing …