Image for post
Image for post

In the attached video I discuss the following topics:

  1. Save links and images from the webpage.
  2. Mark URL as complete in the pipeline once it is fully parsed.
  3. Refactor: extract text compressor to the separate function (similar to decompressor).
  4. To-do: define a task for the “webpage data parser” worker.
GZIP compress web page’s content and save in MySQL using GORM (Golang)

Compress using GZIP

We retrieve the web page content as a text body. Since we expect to have many URLs saved locally, it would be optimal to compress this date; thus, it will take less storage. By my benchmarks, we can see an average of 60–85 % compression level. …


Image for post
Image for post

I started this video as a review of my previous day — part of the January Coding Challenge. But in the end, I started covering code review topics for my ETL pipeline for fetching and processing URLs. I also cover how I plan my coding tasks, how I use GitHub issues and projects for keeping track of future and current features.

In this video post, I will cover the following topics.

Coding parts:

  1. Code review of data streaming pipeline for fetching URLs, compressing its data, and saving for further processing.
  2. Use of URL parser to get text, image, link information from the web-page (demo). …

Image for post
Image for post
Photo by Shahadat Rahman on Unsplash

Last month, I worked on the initial version of the data-pipeline for CodeKN.com’s text-processor. At the current version, it fetches URLs, does basic validation for eligible content, and then passes it for further text-analysis.


Image for post
Image for post
How to Build a JSON API with Node.js?

In this video tutorial, my goal is to create an example Node.js backend API which can be used for React / Next.js projects. API will return a JSON response.

The thing is most of the parts of React / Next.js projects require working backend API so you can fetch data. It’s relatively easy to do so in Python or Golang. Usually with a few lines of code. In this coding session, I want to explore for myself if it is the same in Node.js.

Since it is a coding session and a video-post at the same time, on this page I will post the process, which might contain a lot of “progress content.” If you want to see the result and just a short explanation, jump to the end of the page. …


Image for post
Image for post
Photo by Max Duzij on Unsplash

Originally published at https://kananrahimov.com on August 17, 2020.

Golang HTML Tokenizer: extract text from a web page

One thing is to load the content of a web page, and another is to extract some valuable information from it. For the second one, sometimes, it is helpful and needed to get only the textual information. Because most of the time, it is the particular text that we are interested in. And, one of the ways to do it in Golang is to use HTML Tokenizer.

In Go, there is a sub-repository package called html which implements HTML5-compliant tokenizer. …


Image for post
Image for post
Courtesy of nginx.com

NGINX Ingress configuration for DigitalOcean’s k8s cluster.

This is more code snippets rather than a post or an article. I originally published at https://kenanbek.github.io on July 22, 2020.

Step 1: Install nginx-ingress

helm install nginx-ingress stable/nginx-ingress --set service.type=LoadBalancer --namespace <YOUR_NAMESPACE>

Step 2: Ingress configuration

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: codekn-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: codekn.com
http:
paths:
- path: /
backend:
serviceName: codekn-service
servicePort: 8081
- path: /api
backend:
serviceName: codekn-service
servicePort: 8082
- host: dev.codekn.com
http:
paths:
- path: /
backend:
serviceName: codekn-service
servicePort: 8082
- path: /api
backend:
serviceName: codekn-service
servicePort: 8082

Step 3: Service configuration

apiVersion: v1  
kind: Service
metadata:
name: codekn-service
labels:
app: codekn
spec:
selector:
app: codekn
ports:
- name: website
port: 8081
targetPort: 8081
- name: api
port: 8082
targetPort…


Step by step guide to configure TLS certificate issuer using Let’s Encrypt on a kubernetes cluster. As an example I use DigitalOcean’s managed kubernetes cluster. In this post you can find instructions on how to configure NGINX ingress controller.

Image for post
Image for post

I originally published at https://kenanbek.github.io on July 27, 2020.

Step 1: Create a namespace

kubectl create namespace cert-manager

Step 2: Install cert-manager

kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.12.0/cert-manager.yaml

Step 3: Create a certificate issuer

apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: mrkenanbek@gmail.com …

Image for post
Image for post
Photo by Christian Wiediger on Unsplash

For one of my projects, I am periodically analyzing GitHub data. One of my search alerts is Kubernetes and AI-related items. And, that’s how I found this great article, “My exciting journey into Kubernetes’ history” (link below). One part of the article was especially interesting to me. It shows an insight of the top 25 Kubernetes contributors.


Image for post
Image for post

When I first discovered the fact that ETCD is written in Go, I was quite surprised. My expectation was C or C++, maybe Python, but not Go. I knew that Go is quite popular in a cloud-native word and was aware of the fact that Docker and Kubernetes are written in Go. But for database systems, you usually do not expect to see new languages in use (here I should say relatively new, because first release of Go was back in 2009).

That’s why I decided to invest some time in the future and check popular database implementations in Go. The most recent example is Dgraph Graph Database. I use it in one of my “research” projects and wrote an introduction article for Go. I also have experience in using ETCD and Prometheus from Go. This article is a kind of road map for myself also to check out other databases written mostly in Go. …


Image for post
Image for post
Pink academical Gopher :)

Hello coders! In this tutorial, I will cover the following topics:

  1. How to setup dgraph? This part will include two installation options:
    Option 1: Install dgraph using Docker
    Option 2: Install dgraph on Kubernetes cluster
  2. How to use dgraph Go client and write to a database?

Before we start, let me quickly explain what Dgraph is. Dgraph.io is a Graph Database written entirely in Go. As per official website it’s distributed by design and has various cool features like ACID transactions and shard re-balancing. It is open source and available under Apache 2.0.

1. How to install dgraph?

Use Docker image locally and run on your local machine:

docker pull dgraph/dgraph:latest

# You can test that it worked fine, by running:
docker run -it dgraph/dgraph:latest…

About

Kanan Rahimov

Software Engineer. https://kananrahimov.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store