Pak Jiddat Website

Software Development - Quick Tips

Thu, 28 Nov 2024 00:00:00 GMT

Following are some quick tips related to Software Development:

Removing flickering of controls in C# windows forms

In windows forms, when form controls are dynamically resized, they may flicker. The following script can be used to remove flickering of form controls:

SetDoubleBuffered(formControlVariable);

public static void SetDoubleBuffered(System.Windows.Forms.Control c)
{
    if (System.Windows.Forms.SystemInformation.TerminalServerSession)
        return;
    System.Reflection.PropertyInfo aProp = typeof(System.Windows.Forms.Control).GetProperty("DoubleBuffered", System.Reflection.BindingFlags.NonPublic
        | System.Reflection.BindingFlags.Instance);
    aProp.SetValue(c, true, null);
}

protected override CreateParams CreateParams
{
    get
    {
        CreateParams cp = base.CreateParams;
        cp.ExStyle |= 0x02000000;
        return cp;
    }
}

Programmatic navigation of user interface elements in Unity

Tue, 26 Nov 2024 00:00:00 GMT

Introduction

Tab Navigation is a Unity project that demonstrates how to move focus on user interface elements programamtically. Currently text boxes and drop downs are supported. The project is based on the EasyTab project. The project contains two Unity scenes, which demonstrate the project.

Benefits

Unity does not have a built in solution for moving focus on user interface elements programmatically. For example if we need to move focus from one text box or drop down to the next by pressing a button, Unity does not provide support for this. It is possible to implement programmatic navigation of user interface elements in Unity but it requires some effort. Programmatic navigation of drop downs requires further effort.

The Tab Navigation package allows programmatic navigation of text boxes as well as drop downs. It would be useful in applications that do not support keyboard and which require changing focus on the click of a button. See the above screenshot.

Limitations

The Tab Navigation project has not been tested with Unity 6.

Features

The Tab Navigation project provides scripts that allow forward navigation, reverse navigation and navigating over the elements of a drop down box.

Installation

Go to EasyTab Unity package release page. Download the Unity Package file. e.g EasyTab-1.3.1.unitypackage
Create a new Unity project
Import the Unity Package file. (right click Assets. import package -> custom package)
Clone the Tab Navigation source code repository
Create Scripts folder inside your project and import the files: CustomDropdown, DownArrow, Enter, ForwardNavigation, ReverseNavigation and UpArrow from the Assets -> Scripts folder of the repository

Usage

Set input field that should have focus at the start. This can be done by clicking on EventSystem and then dragging the input field over the “First Selected” property
Create the user interface for your scene. It should consist of text boxes, drop downs and navigation buttons. See the above sceenshot
Attach EasyTab script to all navigation buttons. Set “Selectable Recognition” property to “As not selectable” from the script properties
Attach the imported scripts to the navigation buttons. i.e DownArrow.cs, Enter.cs, ForwardNavigation.cs, ReverseNavigation.cs and UpArrow.cs
Attach CustomDropdown.cs script to all the drop downs in the scene
Set the selectable color of each text box and drop down from the properties inspector. This step is optional
The next and previous Selectable can be configured for each text box and drop down. To configure the next and previous Selectable, Attach EasyTab script to the drop down or text box. Set the Jumping Policy to “Use only jumping” and then drag the drop down or text box to the Next Jump and Reverse Jump fields

Licence

The Tab Navigation project is available under the MIT licence. You may use the application source code in your own projects.

Production and Cost

Wed, 24 Nov 2021 00:00:00 GMT

By Nils R. Barth - Self-made in Inkscape.This W3C-unspecified vector image was created with Inkscape ., Public Domain, https://commons.wikimedia.org/w/index.php?curid=3497302

Introduction

I recently took the Microeconomics course on Coursera and found it useful. As part of the course, we were given some practice questions related to fixed costs and variable costs. In this article I will describe fixed costs and variable costs.

According to Investopedia Variable Costs are costs that change in proportion to how much a company produces or sells. In contrast, Fixed Costs do not change with an increase or decrease in the volume of items produced or sold. Fixed costs have to be payed by the company irrespective of any business activity.

What are fixed inputs and variable inputs for different industries

Background

The relative size of the fixed costs to variable costs varies considerably for different industries. It is important for a manager (or owner) to understand the differences between these cost categories. This discussion topic is designed to allow you to get better at understanding this difference.

Discussion Question

Consider 3 types of businesses/industries: 1) computer manufacturing, 2) computer software; and 3) pizzerias. What is the relationship between fixed costs and variable costs for each of these industries? Which 1 of these 3 industries has the largest share of fixed costs as a percentage of all costs?

Answer

Computer Manufacturing

For the Computer Manufacturing industry fixed costs are those costs which do not change with the number of computer parts that are produced or sold. Variable costs are costs which change with the volume of computer parts that are produced or sold.

Fixed costs may include rent, mortgage and lease payments for the production plant, salaries of staff that do not change with production such as manager salaries. Insurance and property taxes may also be regarded as fixed costs.

Variable costs include cost of labor, commission on sales of the computer parts, cost of raw materials used in production and utility payments such as electricity bills. Cost of packaging and depreciation of plant equipment are also variable costs.

Computer Software

For the software industry, fixed costs are those that do not change with the volume of software that is produced or sold. Variable costs include costs that change with the volume of software that is produced. The volume of software that is produced by software businesses depends on the number of software developers and measures taken to improve developer productivity.

Fixed costs would include license cost of operating systems, utility costs such as electricity bills, salary of managers. If the software is being developed in house, then payment of property taxes, office rent, office lease, insurance costs of computer equipment are also examples of fixed costs.

Variable costs would include the salary of software developers, cost of computer equipment, software licenses and cost of software and online services that increase developer productivity.

Pizzerias

Pizzerias are businesses that make pizzas. The fixed cost for Pizzerias are costs that do not change with the volume of pizzas that are produced. It would include rent, mortgage and lease payments for the kitchen where pizzas are made. The loans taken for payment of kitchen equipment. Salaries of managers. Payment of utility bills is also fixed because it does not change with the number of pizzas that are produced.

Variable costs for Pizzerias are costs that change with the volume of pizzas that are produced. Variable costs include salaries of cooks, cost of raw materials used to produce pizzas such as bread, cheese, cooking oil, vegetables, meat etc.

I think the computer manufacturing industry has the largest share of fixed costs as a percentage of all costs. Computer manufacturers in most cases can easily increase the number of units produced. This would not be possible if variable costs were high.

What are the fixed costs and variable costs of MOOCs

Background

Companies like Coursera are dramatically changing higher education by offering good quality courses to the masses for a very low cost. But a more interesting question is why did it take so long? Perhaps the answer lies in the way technology has changed the fixed and variable costs of offering education in the last 10 years.

Discussion Question

What are the fixed and variable costs of offering a Coursera course? In what way have these costs changed during the last 10-15 years? How do these fixed and variable costs compare to the costs of offering a traditional face to face course for a typical 4-year college?

Answer

Massive Online Open Courses (MOOCs) is basically a video streaming provider. It may also offer software as a service, such as online software labs or applications. MOOCs provide a virtual learning environment at a very low cost.

Over the last 10-15 years there has been major improvements in computer technology. Computer parts such as Hard Drives and Processors cost much less and have much better performance. This has decreased the variable costs of providing online courses.

A MOOC that sells hundreds of courses online can sell thousands of courses without incurring a large variable cost. The same computer equipment can be used to serve many more courses.

This would mean that the cost of computer equipment and electricity is a fixed cost since it does not change with the increase or decrease in the number of courses that are offered. So I think over the last 10-15 years the fixed costs of offering a Coursera course has increased while the variable costs have decreased.

Typical colleges provide a physical learning environment at a high cost. Both MOOCs and colleges provide courses to students but at very different rates. Businesses that provide MOOCs and colleges have different types of fixed and variable costs.

For MOOCs and colleges, fixed costs are costs that do not change with the number of courses that are offered. Variable costs are costs that change with the number of courses that are offered.

For MOOCs fixed cost includes rent, mortgage and lease payments for the office buildings and manager salaries. Cost of video streaming equipment, cost of servers that provide software as a service and the license cost for software are also fixed costs.

For colleges the fixed costs would include rent, mortgage and lease payments for the college buildings and salaries of certain staff such as the college management.

For MOOCs, variable costs includes staff salary and payment to colleges that provide the online course material. The MOOC business may have its own data center. It may also use a cloud computing provider such as Google Cloud Platform or Amazon Web Services.

If the business has its own data center, then its variable cost may include the cost of running the data center. This would include utility costs, depreciation cost of data center equipment, license costs for the software and equipment. If the business uses a cloud computing provider, then its variable costs will include the subscription fees paid to the cloud computing provider.

For colleges variable costs are costs that change with the number of courses that are offered. This includes teacher salaries, cost of providing class rooms, utility costs, recreational services and facilities offered to students.

How long is the long run

Background

How long the long run is will vary from industry to industry.

Discussion Question

How long would you estimate the long run is in the following industries

The market for soda sold from street carts in the Chicago Downtown district.
The market for meals at newly trendy Sushi restaurants.
The market for electrical engineers.
After 1999, the market for movies that are suspiciously similar to The Matrix?

Answer

The market for soda sold from street carts would exist for many years. Changes in the climate or customer preferences may cause the market to shutdown.
The market for meals at new trendy Sushi restaurants may exist for many years. Increase in fish prices or competition from other types of restaurants may cause Sushi restaurants to shutdown.
The market for electrical engineers is likely to remain for a very long time. Currently society depends a lot on electricity and there is a large market for electrical engineers.
The market for movies that are similar to Matrix would likely remain for a few years after the release of the Matrix movie. After that the theme of the movie may become outdated and the market may shutdown.

What are different production functions for different firms

Background

Different firms have different production functions.

Discussion Question

Provide a brief description of the production function for each of the following firms. What is the firm’s output? What input does it use? Can you think of any special features of the way production takes place in a firm?

An Iowa wheat farm
An Arizona vegetable farm
A local arc-welding farm
Apple Computers
Macy’s

Answer

Its input would include cost of farming machinery, Land where the wheat is produced, Labor that is used in production, Fertilizer, seeds, insecticides, and water that is used to grow wheat. Storage facilities used to store the harvested wheat.
The output would be harvested wheat. This output goes through different industries that add value such as industries that produce baked items like bread, cake, biscuits etc.
Its input would include cost of farming machinery, Land where the vegetables are produced, Labor that is used in production, Fertilizer, seeds, insecticides, and water that is used to grow the vegetables. Storage facilities used to store the harvested vegetables.
The input for Apple Computers would be the raw materials that are used to produce the computers. Such as plastic, silicon, metals. Manufacturing equipment that produces computer parts from the raw material. Electricity for the manufacturing equipment. Laborers that check for defective items. The output is the Apple Computer.
Macy’s is an American retail departmental store that offers several categories of products. The input for Macy’s would be the products that are provided by whole-sellers. The buildings used as retail shops, employees and utilities used by the shops such as electricity are also inputs. Order processing systems that include point of sales equipment, human resource management systems, customer management systems, business process management systems are also inputs. The output is the final packaged products that are sold to customers.

Why didn’t the introduction of computers have an effect on labor productivity between 1973 and 1995

Background

Economists have intensively studied productivity trends during the past 50 years in an effort to identify factors that may have contributed to improvements. One of the major puzzles in this research is that productivity growth seems to have slowed down just as computers were coming into more widespread usage in the U.S. economy. From 1959 to 1973, average labor productivity increased at an annual rate of nearly 3 percent per year and total factor productivity growth was more than 1 percent per year. During the following 2 decades, however, both rates of productivity increase slowed dramatically. What is odd about this finding is that these 2 decades were characterized by the rapid introduction of computers into practically all areas of the economy. Presumably these actions should have increased productivity.

After 1995, productivity performance in the U.S. economy improved dramatically, and this is when the effect of computers began to appear.

Discussion Question

Why do you think the introduction of computers did not have an effect on labor productivity in the U.S. between 1973 and 1995? Please explain. Exactly how does computer technology increase productivity?. Who experiences the gains in productivity spawned by computers?

Answer

I think computer technology improves productivity by improving the information flow. Workers have access to more information and can make better decisions based on facts. Also computer technology has improved the quality of products that are produced. Furthermore computer technology has increased automation in production. I think all these factors have led to an increase in worker productivity.

I think the main benefit of increase in productivity is experienced by the company managers and owners who make more money. Products can now be produced at a much lower cost than before.

I think between 1973 and 1995 computer technology was not very advanced. Computers could not perform many tasks and their practical use was not well understood. It was a new technology at that time and society was not used to it. Communication and collaboration tools did not exist. Computer hardware did not have good performance and so the software that was based on it could not do much. Also computer algorithms were newly developed and their practical applications were not well understood.

By 1995 computers hardware had improved a lot in terms of performance. Also computer software became more complex with more features.

Strange as it may seem, I think society is still not quite used to computers. Computers have created many opportunities and markets, but also I think the use of computers is sometimes unnecessary.

Elasticity of demand

Mon, 16 Aug 2021 00:00:00 GMT

Income elasticity of demand for inferior goods. Wykis, Public domain, via Wikimedia Commons

Introduction

I recently took the Microeconomics course on Coursera and found it useful. As part of the course, we were given some practice questions related to elasticity of demand. In this article, I have listed these practice questions and my answers to the questions.

The concept of elasticity of demand is a measure of how responsive the quantity demanded of an item is to a change in some metric such as price or income. The concept of elasticity has many applications in Economics. For example calculation of total revenue, effect of taxes of buyers and sellers etc.

Price elasticity of demand

According to Wikipedia, the Price elasticity of demand is defined as: “A good’s price elasticity of demand (PED) is a measure of how sensitive the quantity demanded of it is to its price. When the price rises, quantity demanded falls for almost any good, but it falls more for some than for others. The price elasticity gives the percentage change in quantity demanded when there is a one percent increase in price, holding everything else constant. If the elasticity is -2, that means a one percent price rise leads to a two percent decline in quantity demanded. Other elasticities measure how the quantity demanded changes with other variables (e.g. the income elasticity of demand for consumer income changes)“.

Income elasticity of demand

According to Wikipedia, the Income elasticity of demand is defined as: “In economics, the income elasticity of demand is the responsivenesses of the quantity demanded for a good to a change in consumer income. It is measured as the ratio of the percentage change in quantity demanded to the percentage change in income. If a 10% increase in Mr. Ruskin Smith’s income causes him to buy 20% more bacon, Smith’s income elasticity of demand for bacon is 20%/10% = 2”.

Cross elasticity of demand

According to Wikipedia, the Cross elasticity of demand is defined as: “In economics, the cross elasticity of demand or cross-price elasticity of demand measures the percentage change of the quantity demanded for a good to the percentage change in the price of another good, ceteris paribus.[1] In real life, the quantity demanded of good is dependent on not only its own price (Price elasticity of demand) but also the price of other “related” products”.

Should the U.S. federal government decriminalize the use of marijuana for recreational purposes

Background

In the U.S. one of the most contested issues of the past 30 years has been how to deal with the issue of drug-related problems in this country. In fact, we introduced this problem in Module 2. Most of the efforts to deal with the problem so far have been by reducing the supply (mostly through enforcement) and the so-called “war on drugs”. But this war on drugs has resulted in very large expenditures in the criminal justice system. And many argue this policy has had very limited, if any, effect on reducing the use of illegal drugs.

At the center of the debate has been the use of marijuana, a product derived from the cannabis sativa plant. While some continue to argue for maintaining the same policy, a strong movement has been calling for a different approach. Alternatives include dealing with the issue as a public health issue, not an enforcement issue. This approach basically would target demand instead of supply. Nevertheless, many have argued for the complete decriminalization of marijuana.

This debate has taken the front pages in the last few years as many states have passed legislation legalizing the use of marijuana for both medical and recreational purposes.

Here is some useful background material.

In favor of legalization:

Gary S. Becker, “The Failure of the War on Drugs” James Ostrowski, “Thinking About Drug Legalization” Against legalization:

Robert L. Maginnis, “Legalization of Drugs: The Myths and the Facts” Drug Enforcement Administration, “Response to JAMA Article Titled ‘Marihuana as Medicine’” No opinion but useful:

Jenny Williams, Rosalie Liccardo Pacula, Frank J. Chaloupka, and Henry Wechsler, “Alcohol and Marijuana Use Among College Students: Economic Complements or Substitutes”

Discussion Question

Should the US federal government decriminalize the use of marijuana for recreational purposes?. You must limit your response to concepts of the price elasticity of demand for marijuana and, for purposes of this discussion, completely refrain from any moral or ethical arguments.

Answer

I think decriminalizing the use of marijuana for recreational purposes will make the use of marijuana in society common. I think this will improve public knowledge about the negative affects of marijuana and will lead to an eventual decrease in the consumption of the drug.

From the point of view of economics, decriminalizing the use of marijuana for recreational purposes will increase the supply of the drug in the market. This will cause a downward shift of the supply curve, which will decrease the equilibrium or market price of marijuana. A decrease in the price will reduce the incentive for sellers to sell marijuana. A decrease in price should not affect the quantity consumed by much, since the demand for marijuana is inelastic. A decrease in the market price of marijuana does not affect buyers much. It should decrease the burden on the criminal justice system.

What are the policy implications of the price elasticity of demand for abortions

Background

Using econometric techniques, Marshall Medoff estimated the price elasticity and the income elasticity of demand for abortions in the U.S. to be -1.07 and 1.24 respectively (see source below). To calculate these elasticities he used state data pooled over the census years 1982, 1992, and 2000.

Source: Medoff, Marshall H., “Price, Restrictions and Abortion Demand,” Journal of Family and Economic Issues, 28(4), December 2007:583–599.

Discussion Question

Use the elasticities found by Medoff to answer the following 2 questions. Again, you must limit your response to concepts of elasticity of demand for abortions and, for purposes of this discussion, completely refrain from any moral or ethical arguments.

What are the economic and policy implications of the price elasticity of demand for abortions to further state and/or federal restrictions on abortions? What are the implications of the income elasticity of demand for abortions when one considers the accessibility of abortions between low- and high-income women?

Answer

The data collected by Marshall Medoff shows that the absolute values for the price and income elasticities for having abortions is more than 1, which means elastic. So a small change in cost for abortions or the income of users involved in abortions will lead to a large change in number of abortions.

States that want to place restrictions on abortions may want to increase the cost of having abortions. A small increase in cost of abortions should not affect hight income users. To decrease abortions fairly among all income classes, restrictions on abortions may involve increase in the cost of abortions in proportion to the income of the users.

Low income women are less able to afford abortions and are more affected by changes in price of abortions. The income elasticity of demand for abortions is most likely higher for low income women as compared to high income women.

Which items are best to highlight on storefronts

Background

Have you ever wondered how stores decide which discounted items to highlight on their marquee?. A drug store may choose to highlight a discount on milk, while a gas station may choose to highlight a discount on diet soda. But each of these stores sells thousands of different items; why did they highlight these 2 items?

Discussion Question

How could the use of the concept of elasticity answer this question of which items are best to highlight on storefronts?. Please give specific examples in answering the question.

Answer

Stores will want to highlighted discounted items with high elasticities of demand. This is because for items with a high elasticity of demand, a small decrease in price causes a large increase in quantity demanded. Consumers of elastic items will be more interested to know about a discount in the price of these items. Hence it makes sense to highlight discounted items with high price elasticities of demand.

Items that are non essential and have substitutes usually have a higher elasticity of demand. For a drug store, medicines are essential items, while milk is a non essential item with substitutes such as soft drinks or bottled water.Drug stores would want to highlight discounted items that have a high price elasticity of demand. This will increase the sales of that item as compared with items that have a lower price elasticity of demand.

A gas station’s main purpose is to sell gas. People who visit gas stations mainly want to buy gas. A gas station would not benefit much from highlighting discounts in prices of essential items, because consumers of essential items will most likely buy items, even in the case of price changes.

A Gas station would benefit more by highlighting non essential items that have a high price elasticity of demand such as diet soda.

If you want to pay the lowest rent possible, should you hope that demand for apartments is elastic or inelastic

Background

The price elasticity of demand determines which side of the market—buyers or sellers—have more leverage when negotiating the rent of apartments. This is an important issue to understand for any college student choosing to live outside of college dorms during his/her time in college.

Discussion Question

Suppose that in your college town real estate developers are building thousands of new student-friendly apartments close to campus. If you want to pay the lowest rent possible, should you hope that demand for apartments is elastic or inelastic?. Please explain and use a specific example.

Answer

If the demand for apartments is elastic, then a small change in price results in a large change in quantity demanded. When this happens landlords are less likely to increase prices because people are more likely to move to other apartments in case of price increase. Hence people who want to rent apartments have leverage.

If the demand for apartments in inelastic, then a change in price does not result in a large change in quantity demanded. This usually happens when apartments are in great demand and students have no choice but to rent specific apartments. When this happens students who want to rent apartments have less leverage.

So students who want to rent apartments will want to hope that the demand for apartments in inelastic.

Price elasticity of demand and the market for tablets.

Background

Say you were advising Tim Cook (CEO of Apple), who was considering decreasing the price of the iPad by 30% (in order to compete with Amazon’s new Kindle). Further, assume you knew the price elasticity for iPads was -1.3.

Discussion Question

What would be the % change in the quantity of iPads sold as a result of the reduction in price?. What would be your advice to Tim Cook?. If you did not know the price elasticity of iPads and would have to guess, would you guess the demand for iPads was price elastic or inelastic?. Please explain your answer using a specific example.

Answer

I think the demand for iPads is elastic because consumers have many options to choose from. Decreasing the price of iPads should increase the quantity sold by -1.3*-30 = 39%.

When the price of an item increases, the total revenue increases if the price elasticity of demand of the item is inelastic. This is because an increase in price results in a smaller decrease in quantity demanded. Since price increases more than quantity demanded, the total revenue increase.

But if the price elasticity of demand in elastic, then the total revenue decreases as a result of an increase in price.

Since the demand for iPads is elastic, decreasing the price increases the quantity demanded by an amount that is larger than the increase in price. Since quantity demanded increases more than the price decreases, the total revenue increases.

What would be the effect of a beer ban on campus during certain holidays

Background

Beer tends to be one of the most inelastic goods in the United States. An average of many studies suggests the price elasticity of demand for beer in the U.S. to be around -0.26.

Discussion Question

Suppose the city officials around a college town are considering banning the sale of beer during certain holidays in order to prevent student misconduct from intoxication. Given the price elasticity of demand for beer, what would be the effect of such a ban? Please explain using examples.

Answer

If a product has an inelastic demand, it means that its quantity demanded does not change much, when the price changes. This is usually the case for products that are considered essential or addictive, such as beer.

Cutting the supply of an item that is considered as essential, will result in an increase in the price of the item. Low income students will not be able to afford beer. This may result in an increase in the number of students with mental health related problems and also possibly an increase in the crime rate.

Creating Snaps for PyQt5 applications

Tue, 10 Aug 2021 00:00:00 GMT

Introduction

Snap Apps or Snaps are applications for the Linux platform. They are managed using the Snappy package management system. Applications implemented using the Snap format can be installed across a wide range of Linux distributions.

Snaps are currently supported by Arch Linux, CentOS, Debian, Fedora, Solus, Manjaro Linux, Linux Mint, OpenEmbedded, Raspbian, OpenWrt and openSUSE.

AppImage and Flatpack, which are similar to Snaps also allow portable distribution of software.

Snaps are not dependent on a specific apps store. They may be downloaded from any source. Ubuntu Snap Store is the main source for snaps. Many popular Linux applications such as Octave, Skype, Stellerium, GIMP, LibreOffice, Blender etc can be downloaded as Snaps from the Ubuntu Snap Store.

In this article I will describe how to package and distribute a PyQt5 application as a snap app. The PyQt5 application is first converted to executable format using the pyqtdeploy tool. See the article: Creating desktop applications with Python using PyQt5, on how to create executable files for PyQt5 applications.

It is possible to package a PyQt5 graphical application directly without converting to executable file format. However this requires knowledge of the package dependencies and desktop platforms. A self contained executable file that contains all the package dependencies such as Python interpreter, Qt5, PyQt5 etc can be deployed easily as a snap application.

Snap, Snapd and the .snap file format

Snaps can be installed and removed using the snap command. The snap command has several uses. For example:

snap version, shows the version of snap and snapd installed.
snap find “media player” shows all snaps related to “media player”.
snap info vlc, lists information about the vlc snap that is currently installed.
sudo snap install vlc installs the vlc snap from the Snap Store.
snap list displays the list of installed snaps.
sudo snap refresh vlc updates the vlc snap.
sudo snap remove vlc removes the vlc snap.

Snaps are published to channels depending on their development status. Snaps may be published to the following channels: stable, candidate, beta and edge. When installing snaps, using the snap command or a Snap GUI front-end, the channel for the snap may be specified.

Snaps are managed using the snapd tool. Using snapd, we can install, remove, backup and restore snaps. Snaps are automatically updated. They can also be rolled back to previous versions. A snap app is deployed in deltas, meaning that only the parts of the application that have changed are updated.

A Snap app consists of a single .snap file. All applications dependencies are contained within this file. A snap file is based on the squashfs file system. Squashfs is a compressed read-only file system used by containers such as Docker. It allows several directories to be merged and mounted as a single directory. A squashfs file may consist of several directories mounted from different locations. The contents of the file can be viewed using the command:

unsquashfs -l file-name

When a Snap App is installed, it is mounted read only at the location: /snap/snap-name/revision/

Developing Snaps

Snaps are developed using the snapcraft tool. Snaps can be developed on LXC/LXD containers, Docker containers or the host system. The Snapcraft tool has useful documentation on creating Snaps.

The article Create your first Snap is an excellent starting point for getting started with Snap application development. First we need to install snapcraft using the command:

sudo snap install --classic snapcraft

This installs snapcraft as a snap app. The snap command requires the snapd tool. Snapd may be installed on Ubuntu using the command:

sudo apt-get install snapd

Once the snapcraft snap has been installed, we can create a snap using the command:

snapcraft init

This command creates a snap folder in the current directory. The snap folder has a single file called snapcraft.yaml. This is the main configuration file for the Snap App. It contains all information that is needed to build the Snap. The end result of building a snap is a .snap file. This file may be installed locally using the snap command or it may be published to a snap store.

The snapcraft.yaml file

The main parts of the snapcraft.yaml file are as follows:

name. This is the short name of the snap.
base. This is the name of the base snap. The snap being built will use libraries and files provided by the base name. If the base snap is omitted, then snapcraft runs in legacy mode without new features and bug fixes. The base snap may be set to core, which provides an Ubuntu 16.04 base. core18 provides an Ubuntu 18.04 base. core20 provides an Ubuntu 20.04 base.
version. This is the version of the snap.
summary. This is a short summary description of the snap.
description. This is the long description of the snap.
grade. This indicates the development status of the snap. It is set to devel by default. When set to stable, the snap may be published to the stable and candidate channels.
confinement. This indicates the security level required to run the snap. The value devmode provides flexible security level that is useful during development. strict should be used for snaps that are ready to be published. classic removes security restrictions and effectively provides device ownership to the snap.
parts. A snap consists of one or more parts, which describe the application. For example a snap may consist of a server part which provides a daemon service or a GUI part which provides a graphical front-end. Each part contains the part name, the location of the source code for the part and a plugin that describes how the part should be built. For example Python applications are build using the python plugin. A part also contains the dependencies required for building the part.
apps. The apps section defines one or more applications. Each application has a name and a command which gives the location of the executable to run, when the command is entered. A Snap may contain several applications. Applications are run using the command:

snap-name.app-name

If the app name is same as the snap name, then it may be run using the command:

snap-name

Building the snap

After the snapcraft.yaml file has been updated, the command:

sudo snapcraft --debug

should be run from the directory containing the snap folder. This command creates a .snap file using the information given in the snapcraft.yaml file.

Snapcraft uses Multipass to build the snap file. Multipass is a virtual machine management system. It allows launching and managing Virtual Machines on Linux, Mac and Windows. The —debug options causes snapcraft to open the virtual machine command prompt in case of errors. This makes it easier to debug problems.

When using multipass, the first time the snap is built it takes between 15 to 20 minutes to download the virtual machine image and package dependencies. Future builds take much less time.

If the snap is being built from within a LXC/LXD container, then it makes sense to build the Snap on the current system, instead of within a virtual machine running within the container. The —destructive-mode option should be used when building snaps within LXC/LXD containers.

Once the snap has been created, we can install it using the command:

sudo snap install --devmode snap-file-name

Example snapcraft.yaml file for executable files

A sample snapcraft.yaml file for executable files is as follows:

name: snap-name # you probably want to 'snapcraft register <name>'
base: core20 # the base snap is the execution environment for this snap
version: '1.2.0' # just for humans, typically '1.2+git' or '1.3.2'
summary: Package summary # 79 char long summary
description: |
Package description

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

layout:
  source-file-path:
    bind-file: $SNAPtarget-file-path
parts:
  # The application files      
  app-files:
    plugin: dump
    source: path-to-folder
    filesets:
       data-files:
         - data-file-path-1
         - data-file-path-2
       binaries: ["bin/bin-file-name"]
       desktop-files:
         - usr/share/applications/app-name.desktop
       icons:
         - usr/share/applications/app-icon.svg
    stage:
      - $data-files
      - $binaries
      - $desktop-files
      - $icons
    prime:
      - $data-files
      - $binaries
      - $desktop-files
      - $icons
    organize:         
      "data/*.db": "usr/local/share/"  
      "data/*.desktop": "usr/share/applications/"
      "data/*.svg": "usr/share/applications/"
    
apps:    
  app-name:    
    extensions:
      - kde-neon
    command: bin/bin-file-name
    plugs:
    - desktop
    - desktop-legacy
    - wayland
    - unity7
    desktop: usr/share/applications/app.desktop

The above file contains all the information needed to create a snap file for a PyQt5 executable file. It may be used for any executable file that needs to run on Gnome or KDE.

A typical desktop executable application consists of an executable file, desktop launcher file, application icon and data files. All these files should be placed in a single folder. The source entry for the app-files part should point to this folder. In the above example this folder is path-to-folder.

All these files need to be copied to the correct location by the build process. This is done using the dump plugin. The dump plugin simply copies the contents of a file or folder from the location specified under source, to another location.

filesets entry allows us to group all project files. Each entry under filesets is for a specific category of files. This allows us to reference all files in a category using a single key name. For example data-files.

The stage and prime entries are used to specify the files that should be copied over from the build stage. In the above example, we specify that the data files, binary files, icons and desktop launcher files should be copied over.

The organize entry allows us to specify how the application files should be placed in the snap. In the above example, we specify that the data files in path-to-folder/data should appear in usr/local/share.

The layout entry allows us to specify refer to files from within our application code. In the above example source-file-path refers to the path that is used in the source code of our application. This should be an absolute path. The $SNAPtarget-file-path is the actual location of the file in the snap. $SNAP is an environment variable that refers to the base location within the snap.

The usr/share/applications/ folder is a standard location for desktop launcher files. The contents of a sample app.desktop file are:

[Desktop Entry]
Name=App Name
X-GNOME-FullName=App Name
Comment=short-description
Categories=GNOME;category-name (see: https://developer.gnome.org/menu-spec/);
Exec=snap-name.app-name
Icon=${SNAP}/usr/share/applications/app-icon.svg
Terminal=false
Type=Application
Version=1.0.0

The snapcraft.yaml file also needs to specify the desktop environments, in which the GUI app will run. This can be done by listing the plugs required by the app. In the above example, the plugs entry indicates that the app needs to be run on desktop, desktop-legacy, unity7 and wayland desktop environments.

Publishing the snap

Once the Snap has been created and tested, it can be pushed to the Ubuntu Snap Store. In order to publish the Snap to the store, we need to first register an account. Next we need to login to the snap store from the command line using:

snapcraft login

After that we need to register our snap using the command:

snapcraft register snap-name

After that we need to set the grade to stable in snapcraft.yaml. Next we need to rebuild the snap using command:

snapcraft

This will create a .snap file in the current directory. After that we can push our snap to the snap store using the command:

snapcraft push snap-file-name --release=candidate

This command publishes our snap on the candidate channel. We may specify the stable channel once the snap is ready. We should then be able to install our snap from the snap store using the command:

sudo snap install snap-name --channel=candidate

Conclusion

In this article I have described how to create a snap application for a PyQt5 executable file. Snaps are a popular application format. They allow portable distribution of applications. Snaps have several useful features such as interfaces, layouts, hooks and data snapshots. Applications deployed as snaps are secure, easy to maintain and can run on multiple platforms.

Creating desktop applications with Python using PyQt5

Tue, 10 Aug 2021 00:00:00 GMT

Introduction

PyQt5 is a Python package that provides a wrapper for accessing the Qt framework libraries. Qt is a cross platform widget toolkit that allows creating graphical user interfaces. It supports Windows, Android, Linux and Mac. Several options exist for developing GUI applications using Python. For example Pyside, TKinter, PyGTK and Kivy are Python packages for creating GUI applications

The obvious benefit of desktop applications is that they allow the data to remain on the local computer. Also the desktop application is always available since it does not depend on internet connectivity. The drawback of desktop applications is that they are not as easy to access as Web based applications. For instance a desktop application needs to be installed on each computer where it is to run.

Installation

Python virtual environments are often used during development. The benefit of using virtual environments is that they allow packages to be installed independently of other virtual environments or the system wide packages. virtualenv is a virtual environment for Python. To create a new virtual environment using virtualenv, the following command is run:

python3 -m virtualenv path-to-project

will create a folder with standard Python libraries and binaries. This folder works as an isolatedenvironment within which Python applications can be developed. Once a virtual environment has been created, it needs to activated using the command:

source bin/activate

Packages can be installed in the virtual environment using the pip package manager. To install the PyQt5 package, we need to run the command:

pip install PyQt5

To test if the PyQt5 package has been successfully installed, enter the command:

import PyQt5

from the Python shell. If the command does not give errors, then the PyQt5 has been successfully installed

Qt 5 Tools is a package that provides tools for the Qt5 framework. It includes the Qt5 Desginer which is a graphical designer for Qt5 applications. The Qt5 tools package may be installed with the command:

pip install qt5-tools

Creating PyQt5 applications

Python GUI applications are run in the same way as command line applications. The main difference is that instead of output being shown on the console, a graphical window pops up. The main steps in writing a PyQt5 application are:

Import the QtWidgets class using:

from PyQt5 import QtWidgets

Create an instance of the QMainWindow class. Command line arguments are passed to the constructor.

app = QtWidgets.QApplication(sys.argv)

Create an instance of the QMainWindow class. This is the actual user interface class.

MainWindow = QtWidgets.QMainWindow()

Create an instance of the Ui_MainWindow class using:

ui = Ui_MainWindow()

This class is used to setup the user interface class created in the previous step. This class is generated by the pyuic5 tool. The pyuic5 tool is used to generate python code from the user interface class created by the Qt Designer. The following code is used to setup the MainWindow class:

ui.setupUi(MainWindow)

The next step is to create an instance of your own class. As an example the class can be named Ui_Manager. This class is used to setup event handlers for the graphical components. The constructor of this class may take the ui object as parameter.
The next step is to display the window using: MainWindow.show(). This method blocks until the user closes the application.
The final step is to call: sys.exit(app.exec_()). This will end the application

The GUI related code is usually auto generated using the Qt Desginer and pyuic5. We need to keep our custom code such as event handling code separate from the auto generated GUI code. Otherwise the custom code can get overwritten. The benefit of this approach is that it separates the users custom graphics code from the auto generated code. The following code shows the “main” function of a typical PyQt5 application:

if __name__ == "__main__":    
    app        = QtWidgets.QApplication(sys.argv)
    MainWindow = QtWidgets.QMainWindow()
    ui         = Ui_MainWindow()
    ui.setupUi(MainWindow)

    ui_manager = Ui_Manager()
    ui_manager.initialize_ui(ui);
    MainWindow.show()
    sys.exit(app.exec_())

Using the Qt Designer

The Qt Designer allows generating user interface for Qt5 applications. It allows user interfaces to be created by dragging and dropping widgets. It supports several widgets such as ComboBoxes, Lists, Tables, Buttons, Labels and more. It supports four types of layouts which are Horizontal Layout, Vertical Layout, Grid Layout and Form Layout. These layouts allow widgets to be arranged correctly.

Each user interface created in the Qt Designer is saved in XML format as a single file with .ui extension. Each .ui file need to be converted to Python code. This can be done using the pyuic5 tool. For example the following command creates a Python file from a .ui file:

pyuic5 ui-file.ui -o ui-file.py

The generated Python file can be run using python3 ui-file.py. This will run the GUI application. The application should look exactly like the preview in the Qt Designer.

The PyQt5 framework provides classes in Python for creating desktop applications. It provides classes for widgets. For example the QApplication class implements the main window class. Classes such as QLabel and QPushButton implement labels and buttons.

Generating executable files using pyqtdeploy

pyqtdeploy is a tool that allows deploying PyQt applications. It supports deploying applications to desktop platforms and mobile platforms. It supports Linux, Windows, MacOs and Android. pyqtdeploy creates a standalone binary file that can be run on the target platform.

pyqtdeploy can be installed as a Python package using the command:

pip install pyqtdeploy

It requires PyQt5 and Python 3.5 and later. A useful way to learn about pyqtdeploy is to deploy the demo application that is packaged with pyqtdeploy. The demo application is a simple application written using PyQt5 and Python. The documentation for the demo application describes how to build the application on the target platform.

Generating a standalone binary file using pyqtdeploy requires the following steps:

Identify the PyQt modules and other third party components that are required by the application. These components are specified in the sysroot.json file. The sysroot.json file that is provided with the demo application is a good starting point.

The systemroot contains the Python libraries and third part components required by the application. The document Building a System Root Directory describes how to build a sysroot. Building a sysroot can take several hours depending on the components selected. The system root is then built using the command:

pyqtdeploy-sysroot  --sysroot sysroot-linux-64/ --source-dir lib --verbose  sysroot.json

The pyqtdeploy-sysroot command is part of the pyqtdeploy package.

Building a sysroot causes the components specified in sysroot.json to be compiled. Qt5, Python interpreter, Sip and PyQt5 are core components and need to listed in sysroot.json. SSL and Zlib are optional components.

Compiling Qt5 requires certain packages. The article Qt for X11 Requirements describes the dependency packages that are required for building Qt5 from source. For example on Ubuntu the dependencies may be installed with the command:

sudo apt-get build-dep qt5-default

Note that this command requires source entries in the /etc/apt/sources.list file.

The next step is to create a project file that identifies the application source code and all the components used by the application. The project file has a .pdy file extension. It can be created using the pyqtdeploy command. This command opens a graphical application that allows generating the project file.

For example the command: pyqtdeploy pyqt-demo.pdy allows editing project file for the demo application. The document Creating a pyqtdeploy Project describes how to create a pyqtdeploy project.

The next step is to freeze the Python modules and generate a .pro file. This can be done using the command:

pyqtdeploy-build --sysroot sysroot-linux-64/  --verbose pyqt-demo.pdy

The article Building the Application describes how to do generate the .pro file.

The next step is to generate the MakeFile using the qmake command. This can be done using the command:

[path-to-qmake] pyqt-demo.pro

This command should be run from the target build directory which was generated during the previous step. The path-to-qmake is the absolute path to the qmake binary file within the sysroot. This is usually sysroot-linux-64/Qt/bin/qmake. Once the MakeFile has been generated, the next step is to generate the executable file using the command: make.

Conclusion

In this article I have described my experience with creating graphical applications using the PyQt5 framework and deploying the graphical applications using pyqtdeploy.

Islam Companion

Tue, 10 Aug 2021 00:00:00 GMT

Introduction

The Islam Companion project is a desktop application. It is availabe as a snap application on Ubuntu Store. It allows users to read Holy Quran and Hadith. It provides two applications. A Hadith reader and a Quran reader.

Features

The Quran text can be read in 43 languages. These are: Amharic, Arabic, Azerbaijani, Bosnian, Bengali, Bulgarian, Amazigh, Czech, German, Divehi, Spanish, English, Persian, French, Hindi, Hausa, Indonesian, Italian, Japanese, Korean, Kurdish, Malayalam, Malay, Dutch, Norwegian, Portuguese, Polish, Russian, Romanian, Swedish, Somali, Sindhi, Albanian, Swahili, Turkish, Tajik, Tamil, Tatar, Thai, Uzbek, Urdu, Uyghur and Chinese..

Hadith text can be read in Urdu, English and Arabic languages.

Limitations

The Islam Companion has the following limitations:

The text for some languages may be difficult to read because of font size.
The hadith text in English language is a rough translation and may contain grammatical errors.

Installation

The Islam Companion application may be downloaded from the Ubuntu Snap Store.

Development

For details on how to develop the Islam Companion application refer to the GitHub repository for the project.

Acknowledgements

The purpose of the Islam Companion project is to allow users to read the Holy Quran and Hadith text in their own language.

The following resources were used in the development of the project:

Translations of the Quran were obtained from: https://tanzil.net/trans/
Translations of the Hadith were obtained from: http://www.hadithcollection.com/downloads.html
Icons were obtained from: https://www.iconfinder.com/
Banner image was obtained from: https://pixabay.com/
Language fonts were obtained from:
- Persian and Uyghur: https://fontlibrary.org/en/font/xb-shafigh
- Sindhi: https://fonts2u.com/zaheer-sindhi.font
- Amharic: https://www.metaappz.com/References/Amharic_Fonts.aspx
- Divehi: https://www.fontspace.com/mv-dawlatulislam-font-f22932
- Kurdish: https://www.kurdfonts.com/font-info/609
The quran and hadith readers were developed using the PyQt5 framework. The applications were developed in Python language and use Sqlite3 for the database.

To report bugs or make feature requests, post an issue. Constructive feedback is most welcome !.

Licence

The Islam Companion project is available under the MIT licence. You may use the application source code in your own projects.

Producer and Consumer Surplus

Thu, 29 Jul 2021 00:00:00 GMT

By User:SilverStar - Own work, CC BY 2.5, https://commons.wikimedia.org/w/index.php?curid=1450405

Introduction

I recently took the Microeconomics course on Coursera and found it useful. As part of the course, we were given some practice questions related to producer and consumer surplus. In this article, I have listed these practice questions and my answers to the questions.

Producer and consumer surplus are used by economists to measure the total wealth of a society in a free market. In a free market trade is governed only by the principles of supply and demand. In this article I will describe what is meant by producer surplus, consumer surplus, total surplus, price ceiling and price floor. I will describe these terms along with examples and some related questions.

Consumer surplus

According to the Wikipedia: consumer surplus, is the monetary gain obtained by consumers because they are able to purchase a product for a price that is less than the highest price that they would be willing to pay.

The consumer surplus can be illustrated on a supply and demand curve. It is the area below the demand curve that is above the horizontal line that indicates the equilibrium price.

Producer surplus

Wikipedia defines producer surplus as the amount that producers benefit by selling at a market price that is higher than the least that they would be willing to sell for; this is roughly equal to profit (since producers are not normally willing to sell at a loss and are normally indifferent to selling at a break-even price).

The producer surplus is indicated on a supply and demand curve, by the area above the supply curve that is below the horizontal line that indicates the equilibrium price.

Total surplus

Total surplus is the sum of producer and consumer surplus.

Price ceiling

Price ceiling is a government intervention policy that fixes the market price to a value that is less than or equal to the ceiling price. Price ceiling affects the consumer and producer surplus and hence the total surplus.

Price flooring

Price flooring is a government intervention policy that fixes the market price to a value that is more than or equal to the floor price. Price flooring affects the consumer and producer surplus and hence the total surplus.

How can we reduce the shortage of organs

Background

As we discussed in the lessons this module, advances in medical treatments have resulted in a dramatic increase in the number of organ transplants performed each year. A limited supply of organs, however, prevents many individuals from receiving organ replacements that could either save a life or substantially improve the recipient’s quality of life. The Organ Procurement and Transplantation Network (OPTN) reports as of today (2/6/2012) there are 117,011 patients waiting for organs and only 11,663 organ donors.

In the U.S., all states have enacted a variation of the Uniform Anatomical Gift Act of 1968. Under this law, individuals are able to specify that some or all of their body may be donated after their death. The original version of this Act neither allowed, nor prohibited, the sale of human organs. The revised Uniform Anatomical Gift Act of 1987, however, prohibited the sale of human organs

Discussion Question

What type of policies would help shrink (or eliminate) the shortage of human organs? Use the concepts we learned this module to explain your answer.

Answer

The following types of policies can help to reduce shortage of human organs:

Policies that promote development of artificial generation of human organs.
Policies that allow human organs to be traded at a price subject to ethical rules. For example organs may be traded only if the organ donor is not alive.
Policies may be formed regarding occupations that have health hazards. For example coal miners may be given the option of donating their organs in case of death during work.
Policies that allow people to exchange each others organs for free or for a price. An ethical restriction could be that only those needing an organ are allowed to donate one of their other organs.

Should we let price gauging takes place in times of natural disasters

Background

During times of crisis, particularly natural disasters, governments enforce strict rules (and penalties) to prevent price gauging. Price gouging is a pejorative term referring to a situation in which a seller prices goods or commodities much higher than is considered reasonable or fair. So, for instance, sellers of critical necessities such as water and gasoline are not allowed to hike up their prices right after a hurricane.

Nevertheless, as you learned in class, price ceilings tend to reduce social welfare by limiting the number of potential transactions. Someone has a higher incentive to load their pickup with water and drive to a hurricane-devastated area to sell it when they have the chance of charging a higher price for it.

Discussion Question

Do you think we should let price gauging take place in times of natural disasters? Again, please use the models and concepts discussed in the lectures to answer the question.

Answer

I think price gauging should be allowed in times of natural disasters, but it should be subjected to ethical rules. Consider the following two examples:

Example 1

For example in case of a natural disaster the quantity supplied of goods such as rice or flour may decrease. As a result the price of the goods will increase. Many low income consumers may not be able to afford the goods. If the goods are non essential, then the consumers can simple switch to a substitute good.

But if the goods are essential, such as water or some other good for which a substitute is not easily available, then the government should intervene and impose a price ceiling for the goods, so that it can be afforded by all.

Example 2

Consider another example. A disease breaks out in a country and the price of medicine or protective equipment increases. People are willing to pay more for the medicine and protective equipment. Also more people will want to buy. Sellers will sense this and will want to charge more. They should have the freedom to increase the price in order to satisfy the increase in demand for the medicine and protective equipment.

Because of the increase in demand, the demand curve will shift upwards and to the right. Because of this a new equilibrium price and quantity needs to be established, which will be at the intersection of the supply curve and the new demand curve. At this new equilibrium point, the sum of producer and consumer surplus, which is the total economic surplus is maximized.

The amount charged by sellers in many cases is not fair. For example misinformation can cause sellers to over price or under price an item. Also the price charged by sellers may be influenced by other factors such as their individual financial conditions or societal pressures. As a result supplier may not be able to correctly assess the demand for the goods.

As a result the amount charged by sellers may be fixed by local trade unions or the government. For example the government may impose a price ceiling on the sale of the medicine and protective equipment.

Although the intention of the price ceiling is to make the goods available for all, it has the adverse affect that the producer’s surplus is reduced because they are being forced to sell at a lower price. If the seller is not happy, then the total economic welfare is reduced and society as a whole does not gain from the price ceiling.

I think these days the goods consumed by a country are produced all over the world, so there is no direct connection between producers and consumers. The quantity and price of supplied items depends on international trade. Governments can only influence their own producers, but not those of other countries. In case of natural disasters, having good international relations with other countries can be useful.

Why is it so hard to find a cab in high crime neighborhoods

Background

Cab fares are usually regulated in most cities. For instance, in Chicago, where I live, cab drivers are not allowed to charge more than $2.25 for the first 1/9 mile, and $0.20 for each additional 1/9 mile. So, in effect, the city of Chicago imposes a price ceiling on the price of cab fares.

Nevertheless, as you learned in class, price ceilings tend to reduce social welfare by limiting the number of potential transactions.

Discussion Question

Do you think there is a relationship between the price ceiling for cab fares and the fact that it is very difficult to find cabs in high crime neighborhoods? Again, please use the models and concepts discussed in the lectures to answer the question.

Answer

In high crime neighborhoods, it is not easy for people to negotiate a price due to volatile economic conditions. An effective way to maximum welfare in society is to use price ceilings. When a price ceiling is in affect, both suppliers and consumers are aware of the market price before the transaction. There is less chance of disagreement, which can lead to crimes.

Why do we only get peanuts on U.S. flights

Background

People always complain about the lack of complimentary meals in flights. And older people always tell stories of how in the “good old days” airlines used to serve full meals for free on their flights, and they also had more leg room between seats. But, also in the good old days airlines did not have much control over the price they charged for their flights. The U.S. government had strict regulations over the airline industry, including how much they could charge fliers. During those times it was hard for an airline to lower their prices without first getting permission from governmental authorities. So, in effect, the government used to impose a price floor on the market for air fare.

Discussion Question

Do you think there is a relationship between the price floor and the fact that today airlines offer fewer amenities than before? Furthermore, do you think air travel is better now (with fewer amenities) than before? Again, please use the models and concepts discussed in the lectures to answer the question.

Answer

Assuming that the price floor is less than the equilibrium price, having a price floor means supplier can continue to provide the airline service at the same equilibrium price. Hence producer surplus is not affected by the price floor.

Consumer surplus may decrease because some consumers may not be willing to pay at prices equal to or more than the price floor. So demand for airline travel will decrease. To increase demand for airline travel, airlines may adopt strategies such as serving full meals for free and providing more leg room between seats. As a result, consumers will be willing to pay more for airline travel. This may explain why airlines offered complimentary meals on flights in the “good old days”.

I think airlines fares these days is much cheaper than it used to be. As a result consumer surplus is high and there is no need to provide extra amenities on flights such as complimentary meals.

Why would an increase in the minimum wage lead to less unemployment

Background

In class this module we presented the traditional textbook explanation of the minimum wage. The minimum wage is a price floor that tends to increase unemployment by encouraging more workers to enter the labor force to compete for fewer jobs. Nevertheless, this theory has not been so easy to prove by looking at the data. Some studies, in fact, have proven it, but others have not. In fact, some studies have suggested an increase in the minimum wage could lead to less, not more unemployment.

Discussion Question

How could a price floor such as the minimum wage lead to less, not more, unemployment? Again, please use the models and concepts discussed in class to answer the question, and for purposes of this discussion, completely refrain from any moral or ethical arguments.

Answer

Unemployment implies a difference in demand for jobs and jobs available. In other words, there is a difference in quantity demanded and quantity supplied, which means that the market for jobs is not operating at the equilibrium point.

If the current price or wages are below the equilibrium point, then a price floor that is above the current market price but below the equilibrium price will result in a smaller difference in jobs demanded and jobs supplied. Hence there will be less unemployment.

Should you stand in the line or pay someone to do it for you

Background

National Public Radio, NPR, reported the case of a person in China that earns his living by waiting in line for other people.

He is a “professional queuer”. Last year, when Apple introduced its iPad Mini, news reported many instances where some people had paid others to stand in line for them to buy the iPad. For instance, in one high profile case, a man made $1,500 waiting in line to buy the iPad for someone else.

Discussion Question

Do you think these type of transactions increase or decrease social welfare? Suppose the government makes these types of transactions (charging someone to wait in line for them). Which groups in society are bound to lose the most as a result of this law?

Answer

I think these type of transactions decrease social welfare. If the supplier had directly received the additional amount that the consumer was willing to spend, then the supplier would have increase the price of the item, which would increase the supplier surplus. An increase in the supplier’s surplus would increase the total economic surplus.

If the government passed a law making it illegal to pay someone else to stand in line, then this would adversely affect people who are mentally or physically incapable of standing in line, such as disabled people. It will also adversely affect unskilled people who can earn money by standing in line for other people.

Counting problems

Mon, 26 Jul 2021 00:00:00 GMT

Introduction

I recently took the Combinatorics course on Coursera. The course provides useful knowledge about counting and probabilities. I found the course very useful and interesting. In this article I will describe some problems that were given to the students to solve.

Combinatorics is a field of Mathematics that allows us to efficiently count numbers. It includes topics such as probabilities, permutations and combinations. It has applications in many areas outside Mathematics. In this article I will describe some problems related to Combinatorics.

Problems related to permutations and combinations can be grouped as ordered or unordered. They may or may not involve repetitions. There are four main categories of problems. These are ordered with repetitions, ordered without repetitions, unordered with repetitions and unordered without repetitions. For each category there is a formula, which can be used to solve the problem.

Combinations are arrangement of unordered items. They may or may not involve repetitions. Permutations are arrangement of ordered items without repetitions. Tuples are arrangement of items with repetitions.

The formula for combinations without repetitions is n!/k!*(n-k)!, where n is the number of items and k is the number of selections. The formula for combinations is often stated as n choose k.

The formula for combinations with repetitions is (n+k-1)!/(n-1)!*(k)!, where n is the number of items and k is the number of selections.

The formula for permutations is n!/(n-k)!, where n is the number of items and k is the number of selections.

The formula for tuples is n^k, where n is the number of items and k is the number of selections.

Problem 1

How many binary strings of length 6 can be created containing equal number of 0s and 1s.

The binary string of length 6 can be regarded as 6 distinct positions or indexes. These need to be selected 3 at a time. There are 3 selections and 6 items. The indexes are unique so there are no repetitions. Also the selection of the index numbers is unordered. So we can use the formula for combinations without repetitions, which is n!/k!*(n-k)!.

This is equal to 6!/3!*(6-3)! which is equal to 20.

The result can be verified using the following Python script. The script finds all binary numbers of length 6. For each number if the number of 0s and 1s is equal it increases the value of a counter.

def bin_comb():
    """Returns the number of 6 bit binary strings that contain equal number of 
    0s and 1s.
    """
    
    # The number of binary strings of length 6 that contain equal number of
    # zeros and ones.
    count = 0
    # Each number from 0 to 63 is converted to a binary string
    for i in range(64):
        # Convert to binary string
        num = str(bin(i)).replace("0b", "")
        # The number is right padded so the total string length is 6
        num = num.zfill(6)
        # If the number of zeros and ones is equal
        if num.count("0") == num.count("1"):
            count += 1
            
    return count            
        
# The number of binary strings of length 6 that contain equal number of
# zeros and ones.        
count = bin_comb()        
print(count)

Problem 2

How many 6 digit numbers are there with equal number of even and odd digits. Each digit can have a value from 0 to 9.

This problem is similar to the previous problem with two main differences. First each position can be occupied by 5 values. Second a number cannot start with 0.

First we consider the number of 6 digit numbers with 3 even digits and 3 odd digits. This is same as the number of binary strings of length 6 with 3 0s and 3 1s.

So there are 20, 6 digit numbers with equal number of even and odd digits. However for each combination of 3 even and 3 odd digits, each digit can have 5 different values. An odd digit can have the values 1,3,5,7,9 and the even digit can have the values 0,2,4,6,8. So the total number of 6 digit numbers is *20(5^6)**. But this will include numbers that start with 0, which are not valid. So we need to consider the number of 6 digit numbers that start with an odd digit and the number of 6 digit numbers that start with an even digit.

The number of 6 digit numbers that start with an even digit is simply 5 choose 2. This is because, the left most position is even and there are 5 positions left. 3 of these are odd and 2 are even. So the number of 6 digit numbers that start with an even digit is 5!/3!*2!. This is equal to 10. For the left most digit, there are 4 possibilities, which are 2,4,6 and 8. For the other digits there are 5 possibilities for each position. So the total count of all 6 digit numbers that start with an even digit is 10*4*5^5 = 125000.

The number of 6 digit numbers that start with an odd digit is equal to the number of 6 digit numbers minus the number of 6 digit numbers that start with an even digit. This is equal to 20-10 = 10. Each position can have 5 different values. So the total count of all 6 digit numbers that start with an odd digit is 10*5^6 = 156250.

The total count of all 6 digit numbers that contain 3 even digits and 3 odd digits is 156250+125000 = 281250.

This can be verified using the following Python script. The script checks all numbers between 100,000 and 999,999. For each number it counts the number of even and odd digits. If these are equal, then the script increases the value of a counter.


def get_even_odd_count():
    """Returns the count of all 6 digit numbers with equal number of even and
    odd numbers. Each digit can have a value from 0 to 9.
    """
    
    # The number of 6 digit numbers with equal number of even and odd digits
    count = 0
    # All numbers between 1000000 and 1000000 are checked
    for i in range(100000, 1000000):
        # i is converted to string
        s = str(i)
        # The count of even number of digits
        ec = s.count("0") + s.count("2") + s.count("4") + s.count("6")
        ec += s.count("8")
        # The count of odd number of digits
        oc = s.count("1") + s.count("3") + s.count("5") + s.count("7")
        oc += s.count("9")  
        # If the number of even and add digits is equal
        if ec == oc:
            count += 1
            # print(s)
            
    return count
        
        
# The number of all 6 digit numbers with equal number of even and odd digits
count = get_even_odd_count()
print(count)

Problem 3

What will the following program print?

n = 10
count = 0

for i in range(n):
    for j in range(n):
        for k in range(n):
            if i < j and j < k:
                count += 1

print(count)

The above code should print the number of all combinations of i, j and k such that i < j < k, where i, j and k are between 0 and 9.

If we consider the following table:

| i | j | k | total count | | :--- | :--- | :--- | :--- | :---------------------- | | 0 | (1,2,3,4,5,6,7,8) | (8,7,6,5,4,3,2,1) | 36 | | 1 | (2,3,4,5,6,7,8) | (7,6,5,4,3,2,1) | 28 | | 2 | (3,4,5,6,7,8) | (6, 5, 4, 3, 2, 1) | 21 | | 3 | (4,5,6,7,8) | (5, 4, 3, 2, 1) | 15 | | 4 | (5,6,7,8) | (4, 3, 2, 1) | 10 | | 5 | (6,7,8) | (3, 2, 1) | 6 | | 6 | (7, 8) | (2, 1) | 3 | | 7 | (8) | (1) | 1 |

The first column is the possible values for i. The second column contains the possible value for j for the given value of i, such that i < j. The third column contains for each value of i and j, the number of possible values for k, such that j < k. The forth column contains the total number of values for k. It is the sum of the values in the k column. The sum of the total count column is the required result.

The total is 36 + 28 + 21 + 15 + 10 + 6 + 3 + 1 = 120.

The values in the fourth column may be represented with the formula: sigma(1, n+k-2), where n = 10 and k is the position number or the value of i. k has 8 values in total, which is equal to n-2. So the following formula gives the total number of combinations: sigma(1, n-2)sigma(1, n+k-2). Where the sigma functions are nested. This mean that each value for the outer sigma function is a value for k in the inner sigma function. n has the constant value of 10.

This formula can be useful for large values of n. The following Python script uses this formula to find all combinations where i < j < k, for n = 1000.

def seq_sum(n):
    """Returns the sum of the first k natural numbers where k is from 0 to n-2.
    """
    
    sum = 0
    for k in range(1, n-1):
        sum = sum + k*(k+1)/2
    return sum
    
# The value of n    
n = 1000
# The sum of the sequence
s = seq_sum(n)
# The output is displayed
print("The number of combinations that satisfies i < j < k = ", s)

Supply and demand

Sat, 17 Jul 2021 00:00:00 GMT

By Paweł Zdziarski (faxe), Astarot - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=741993

Introduction

I recently took the Microeconomics course on Coursera and found it useful. As part of the course, we were given some practice questions related to supply and demand. In this article, I have listed these practice questions and my answers to the questions.

The principles of supply and demand form the base of modern microeconomics. In this article I will describe the principles of supply and demand along with examples and some related questions.

Law of supply

According to Wikipedia the law of supply states that assuming all other factors are constant, an increase in the price of items supplied leads to an increase in quantity supplied.

Law of demand

According to Wikipedia the law of demand states that assuming all other factors are constant, an increase in the price of items leads to a decrease in quantity demanded.

The supply curve

The supply curve shows the variation of item price (y-axis) and quantity supplied (x-axis). The term “supply” refers to the state of the supply curve and is different from quantity supplied. For example if the supply of an item goes up, then for the same price, the quantity supplied is more. Hence the supply curve shifts to the right. If the supply decreases, then for the same price the quantity supplied is less. Hence the supply curve shifts to the left.

The demand curve

The demand curve shows the variation of item price (y-axis) and quantity demanded (y-axis). This term “demand” refers to the state of the supply curve and is different from quantity demanded. For example if the demand for an item goes up, then at the same price the quantity demanded is more. Hence the demand curve shifts to the right. If the demand decreases, then for the same price the quantity demanded is less. Hence the demand curve moves to the left.

Why is it so hard to find good math and science teachers

Background

One of the major problems in public education in the U.S. is the shortage of science and math teachers. Could we explain this shortage using the supply and demand model?. We might, if we recognize the supply for teachers depends on the opportunity cost for working.

Remember that everything has a cost. The opportunity cost of your time working as a teacher is the salary you could make working somewhere else. Therefore, the larger the salary you could make working somewhere else, the larger your opportunity cost of working as a public school teacher. Salaries in math and science are usually higher than salaries in other disciplines such as humanities and the social sciences. Therefore, a person with a degree in math and science typically gives up a much larger salary to teach than a person teaching in a different discipline.

Discussion Question

Use a supply and demand diagram to illustrate how this higher opportunity cost for math and science teachers creates a shortage of them.

Answer

A higher price implies that there is more competition and teaching standards for math and science teachers goes up and hence there are fewer people who can meet the new standards. As a result there is a decrease in the supply of math and science teachers. This could be prevented by reducing the opportunity cost for math and science teachers.

Where did all the farmers go

Background

In the 1950’s about 17% of the labor force in the U.S. worked on farms. More than fifty years later, in 2004, that percentage was only 3%. What happened? Why did people stop farming in the U.S.? One explanation has to do with increases in productivity in farming.

Discussion Question

Use a supply and demand diagram to explain how increases in productivity in farming creates an incentive for farmers to leave farming.

Answer

Increase in productivity implies improvement in technology. This results in a right shift of the supply curve. It means that at the same price the quantity supplied is more, which implies that the cost of production decreases. A decrease in cost of production means that laborers are payed less. Hence fewer people will want to work on farms.

A lower cost of production implies it costs less to produce and hence the price of items sold by farmers would decrease. A decrease in price would mean farmers will earn less income. Hence they would be encouraged to leave farming.

Why are people moving back downtown

Background

In the U.S. people are moving back downtown. Cities such as Chicago have seen a boom of residential real estate development in their downtown areas. Take a drive around the South Loop (as the south part of Chicago’s downtown is known), and you will see plenty of building cranes. Why is this?

Interestingly enough, one reason may have to do with the increase in online shopping. As people continue to get their shopping done online, large retailers have been closing their stores in the downtown areas. This has increased the availability of real estate in those areas.

Discussion Question

Use a supply and demand diagram to explain how the increase in online shopping has created an incentive for people to move back downtown.

An increase in online shopping would mean there is less demand for retail shops. As a result the prices of items sold at retails shops will increase. This will further discourage people from buying from retail shops.

Fewer retails shops implies more space for buildings such as schools, residential buildings, offices etc. This will encourage more people to move downtown.

An increase in residential buildings usually means a decrease in house prices. A decrease in price results in an increase in quantity demanded. So more there will be more incentive for people to move downtown.

Why would more drug enforcement of illegal drug lords lead to more crime

Background

For the last 40 years, the U.S. has been fighting a war against illegal drugs. The major targets have been marijuana, cocaine, heroine, and, most recently, new drugs such as ecstasy and meth. The goal of this “war” has been to reduce the supply of these drugs coming to the country by either seizing the drugs or by incarcerating those involved in the illegal market. Paradoxically, many argue that this increase in enforcement has led to an increase in the crime associated with the use and distribution of these drugs. Why?

Discussion Question

Use a supply and demand diagram to explain how an increase in drug enforcement could lead to an increase in crime.

Answer

If the supply of drugs decreases, then the supply curve would move left. A decrease in the supply of drugs means people will be willing to pay more for drugs. So the price of drugs will increase. Less users will be able to pay for drugs so people will be more likely to commit crimes in order to pay for drugs.

Why has the price of oil increased in the last 10 years? Or has it

Background

In 1994, the average cost for a gallon of gasoline in the U.S. was $1.21. These days, the same gallon of gasoline goes for about $3.40. On the one hand, we could explain this increase in price to a larger demand for gasoline, particularly from rapidly growing countries such as India and China. On the other hand, if gasoline was, in fact, so expensive, we would see smaller and more efficient cars on the street, but we have not.

I drive a 2011 Honda CRV that gives me an average of about 25 miles per gallon (MPG). The same model in 1995 gave you, are you ready just about the same: 25 MPG. Studies have shown that from 1975 to 1980, the average vehicle weight in the U.S. dropped almost 1,000 pounds, from 4,060 lbs. to 3,228 lbs. By 2005 those reductions had all been gained back, and the average car was as heavy as it was in 1975. If the price of gasoline is higher, why are people driving bigger and less efficient cars?

Discussion Question

Use a diagram of supply and demand to explain why people in the U.S. are driving larger and less fuel efficient cars today if the price of gasoline has continued to increase.

Answer

An increase in cost of gasoline, would mean that it costs more to produce and deliver cars. The quantity supplied for cars will decrease. If fewer cars are being supplied, then in order to consume the available gasoline, the average vehicle weight can be increased. This may explain why people are driving bigger and less fuel efficient cars.

Finding numbers between 0 and 9999

Sun, 11 Jul 2021 00:00:00 GMT

Introduction

I recently took the Combinatorics course on Coursera. The course provides useful knowledge about counting and probabilities. I found the course very useful and interesting. In this article I will describe one of the problems that were given to the students to solve.

Combinatorics is a field of Mathematics that allows us to efficiently count numbers. It includes topics such as probabilities, permutations and combinations. In this article I will describe how to solve a problem in Combinatorics that involves numbers between 0 and 9999.

The problem is to find out how many numbers between 0 and 9999 inclusive contain exactly one “1” and one “3”.

Solution

Numbers containing one “1” and one “3” that are less than 9999, can have 2, 3 or 4 digits. We need to sum the count of 2 digit, 3 digit and 4 digit numbers that contain exactly one “1” and one “3”.

Suppose that we label the digit which does not have the value “1” or “3” as “c”. This digit can have the values 0,2,4,5,6,7,8,9.

Count of two digit numbers

The number of 2 digit numbers that contain exactly one “1” and one “3” are 2, which are 13 and 31.

Count of three digit numbers

For 3 digit numbers that contain exactly one “1” and one “3”, there is only one digit, c that can have values between 0 and 9 excluding 1 and 3. This digit can be the first digit, second digit or third digit.

If c is the first digit, then the second and third digits can be arranged in two different ways, 1,3 and 3,1. For each possible value of c, there are two possibilities for the remaining digits. The possible values for c are 2,4,5,6,7,8,9. i.e there are 7 options. Notice that 0 is excluded because a 3 digit number cannot start with 0. For each of these 7 options there are 2 ways of arranging the remaining digits so there are 7*2=14 3 digit numbers that start with c.

If c is the second digit, then it can have 8 values which are 0,2,4,5,6,7,8,9. For each value of c, there are two possibilities for the remaining 2 digits. So there are 8*2=16 numbers with c as the second digit.

If c is the third digit, then it can have 8 values which are 0,2,4,5,6,7,8,9. For each value of c, there are two possibilities for the remaining 2 digits. So there are 8*2=16 numbers with c as the third digit.

So in total there are 16+16+14=46, 3 digit numbers containing exactly one “1” and one “3”.

Count of four digit numbers

Consider the following table. It shows the different possibilities for the four digit numbers. a and b are digits that can be either 1 or 3. c is a number that can have all values between 0 and 9 except for 1 and 3. The table shows the different ways of arranging a, b and c.

1	3	c	c	number of results
a	b	c	c	(8*8=64 possibilities)
b	a	c	c	(8*8=64 possibilities)
c	c	a	b	(7*8=56 possibilities)
c	c	b	a	(7*8=56 possibilities)
a	c	b	c	(8*8=64 possibilities)
b	c	a	c	(8*8=64 possibilities)
c	a	b	c	(7*8=56 possibilities)
c	b	a	c	(7*8=56 possibilities)
c	a	c	b	(7*8=56 possibilities)
c	b	c	a	(7*8=56 possibilities)
a	c	c	b	(8*8=64 possibilities)
b	c	c	a	(8*8=64 possibilities)

The total is (64*6)+(56*7)=720. There are 720, 4 digit numbers that contain exactly one “1” and one “3”.

The total count is equal to sum of 2 digit count, 3 digit count and 4 digit count. The total is therefore 2+46+720=768.

Code

The following code prints all numbers between 0 and 9999 inclusive that contain exactly one “1” and one “3”. It can be used to verify that the total count is 768.


# Takes as input the upper bound for the numbers.
def number_checker(max):
  vc = 0
  for i in range(max):
    a = str(i)
    oc = 0
    tc = 0
    for j in a:
      if j == "1":
        oc += 1;
      elif j == "3":
        tc += 1
    if (tc == 1 and oc == 1):
      vc += 1
      print(i)
  return vc
  
      
vc = number_checker(10000)
print("The number of integers between 0 and 9999 is:", vc)

Economics in daily life

Sun, 11 Jul 2021 00:00:00 GMT

Ben.rajesh, CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0, via Wikimedia Commons

Introduction

I recently took the Microeconomics course on Coursera and found it useful. As part of the course, we were given some practice questions related to applying economics in daily life. In this article, I have listed these practice questions and my answers to the questions.

According to Wikipedia: Economics is the social science that studies how people interact with value; in particular, the production, distribution, and consumption of goods and services. In this article I will discuss some daily life applications of Economics.

Economics in your daily life

According to Wikipedia: Trade involves the transfer of goods or services from one person or entity to another, often in exchange for money.

Background

Many of the concepts we will be discussing this semester should apply to your daily life, including the ones we talked about this module. This discussion question gives you a chance to begin exploring economics in your daily life.

Discussion Question

Give at least 2 examples of any of the concepts we introduced this module that you observe in your daily life. Try to be very specific describing the examples, and also describe how the economic principle applies to your examples.

Answer

I have tomatoes. I have to eat them every day, so I dont like them very much. My friend has milk cream. She also has to eat it every day and is tired of having it. I like milk cream and my friend likes tomatoes. If I give some tomatoes to my friend and she gives me some milk cream, then we would both be happy with the trade.
I buy the same food from the same shop every few days. I am used to paying the same price for the food. Suddenly the price of one of the food items, x increases by 100%. This is very disturbing because I have a fixed budget and have to allocate money for other routine tasks. What should I do?. I decided to switch from buying the food item x to buying the food item y which is cheaper than x but has the same value in terms of taste. However y is less nutritious than x. I am less concerned with nutrition, so I decided to buy y. y costs about the same as x did few days back from the price increase. So the marginal cost of switching from x to y is small. The cost from switching from x to y is that I lose some nutritional benefits of x.

Visual Economics

Background

A picture is worth a thousand words. I truly believe this is true. Not only can a picture more easily describe a concept, but the process of finding that picture can also be a good learning experience.

Discussion Question

Find 1 picture (photograph) that represents any of the 4 principles we introduced this module. Along with your picture, post a brief description of how this photograph is a good representation of the principle in action.

Answer

The outcome of trade is that both parties benefit from it. The picture illustrates this concept.

What is the opportunity cost of going to college

According to Wikepedia: Opportunity cost is the loss of the benefit that could have been enjoyed if the best alternative choice were chosen instead. Directly or indirectly, opportunity cost underpins the majority of day-to-day economic decisions that are made in society.

Discussion Question

Please list the 2 largest opportunity costs of obtaining a 4-year degree (assume you must go to college full time to get this degree). Now, use 1 of these costs to explain the following trend: When the unemployment rate increases, enrollment in college also increases, and vice-versa.

Answer

You are not able to stay at home and look after family members or socialize with general public. Your circle of friends becomes exclusive.
The money spent on going to college, may be spent on other resources.

When the unemployment rate increases, fewer people are working and more are spending time at home with their families. When people are forced to stay at home with family, they start getting used to being at home and are more willing to go out and explore new opportunities such as going to college.

When enrollment in colleges increases, a lot of people spend a lot of time behind books and in class rooms. In a sense, they become isolated from the rest of society and are not able to participate in activities that add value to society. If value is not being added, then there are fewer opportunities for people, which may result in an increase in the unemployment rate.

When the unemployment rate increases, people are not encouraged to spend on non essential products and services. This may encourage them to spend on college.

When enrollment in college increases, people spend a lot on college tuition, so there is less money for other resources. Because there is not enough money, there are less opportunities for people to work, which can lead to an increase in unemployment.

When Does the Invisible Hand NOT Work

According to Wikipedia: The Invisible Hand is an economic concept that describes the unintended greater social benefits and public good brought about by individuals acting in their own self-interests.

Background

One of the principles we introduced this module is the invisible hand principle: society generates the most welfare when individuals try to maximize their individual interests. As many economists point out, this principle requires good societal institutions in order to work. When institutions do not work correctly, what we have is sort of an invisible foot. So here is the question:

Discussion Question

Please describe a situation in which the invisible hand does not seem to work. Then explain why the principle does not work and what would need to happen for it to work.

Answer

It is common for people to want to increase the amount of money they have in the bank. But if there are no opportunities for spending this money, then it becomes useless. It may even become a burden on society, since interest in payed on the money in the bank account. In many cases, the money in the account holders bank account increases automatically, without the knowledge or explicit consent of the account holder.

The individual may think that his interest is being served, but society as a whole will probably not benefit, because simply transferring money to a users account, without the knowledge or consent of the account holder will not create any value. So it would seem that the invisible hand principle does not work in this case.

To make it work in this case, the banking institutions should transfer money to and from an account after investigating the activities of the account holders. One way to achieve this is to allow the banking transactions to take place, but only finalize the transaction after the activities that led to the transaction have been investigated. The banking institutions should collaborate with other social institutions in this regard.

Another option is to create specialized accounts that serve the needs of different types of account holders. It would be also useful if the banking institutions engage with their customers on a personal level and try to find spending opportunities for their customers. In this way the customers will be happy and society as a whole will benefit because of creation of value, that should result from the customers spending.

What is the Best Way to Teach a Coursera Course

Background

One of the questions we discussed this module was what would be the best way to teach a Coursera course. Since my time is a limited resource, and since there are more than one way of distributing this limited resource, this is a classic example of the economic problem. In our lecture we listed the following three possible distribution schemes:

I teach the class for free to as many students as sign up (this is the way it is done right now).
I teach the class for free to just a few students who pay the most. Assume that either Coursera and/or the University get the money from the participants, while I teach the course “pro-bono” or free.
I choose a few students at random and teach the class to those students for free.

Discussion Question

Choose which 1 of the 3 distribution schemes that you think is the best. Then, clearly discuss why you think this is the best way to distribute the limited resource. In your answer, please also discuss the pros and cons of the 3 distributions schemes.

Answer

Con: If you teach the class for free for all students, then some students may not be motivated to study since they are not paying. Also Coursera and the instructor would not benefit financially, since they are offering a service for free.

Pro: If you teach the class for free for all students, then a lot people benefit from excellent teaching material. This will result in fewer social problems and higher employment. Specially useful in developing countries.

Con: If you teach the class for free to just a few students who pay the most, then a lot of people who dont have the money but are motivated to study will not be able to study.

Pro: Coursera will earn money for their service.

Con: If you teach the class for free to a random students, then not all students will benefit from the class. Some motivated students may be left out. Also Coursera and the instructor will not make any money.

Pro: Some motivated students who are not able to pay the fees will be able to study. But only if they are lucky.

Is it Better to Study in Groups

Background

Many education researchers are discovering the benefits of studying in groups. Yet for an economist, this should have been obvious…or is it?

Discussion Question

Explain the economic benefits of studying in a group versus individually. Please try to offer a specific example to demonstrate your arguments. For instance, would it be better for you to study with another person for the first exam of this class? If so, how many students would be a good number of group partners? Why might the benefits of adding more partners to the group decrease as more partners are added the group?

Answer

The main benefit of studying in a group is that it allows people to share their different views and opinions on a topic. This should improve the students understanding of the topic. I think before studying in a group, the students should first go over the course material individually. Three or four would be a good number of group partners. I think that if more group members are added, students become more distracted since there is a large number of other group members they need to think about. Also students may get over-whelmed if they need to consider too many points of view on a topic.

R Package For Word Prediction

Thu, 10 Jun 2021 00:00:00 GMT

Introduction

Wordpredictor is a R package for developing and evaluating n-gram models. It provides methods for data analysis, data cleaning, data sampling, converting text to tokens, generating n-gram transition probabilities, generating n-gram models and evaluating n-gram model performance.

The goal of the wordpredictor package is to provide a flexible and easy to use framework for generating n-gram models for word prediction.

The package allows generating n-gram models from input text files. It also allows exploring n-grams using plots. Additionally it provides methods for measuring n-gram model performance using Perplexity and accuracy.

The n-gram model may be customized using several options such as n-gram size, data cleaning options and options for extracting tokens from text.

Installation

You can install the released version of wordpredictor from CRAN with:

install.packages("wordpredictor")

And the development version from GitHub with:

# install.packages("devtools")
devtools::install_github("pakjiddat/word-predictor")

The source code repository for the package is on GitHub.

Package structure

The wordpredictor package is based on R6 classes. It is easy to customize and improve. It provides the following classes:

DataAnalyzer. It allows analyzing n-grams.
DataCleaner. It allows cleaning text files. It supports several data cleaning options.
DataSampler. It allows generating sample files of specified size from a text file. It also allows generating train, test and validation sample files from a given input text file.
TokenGenerator. It allows generating n-gram tokens of given size.
TPGenerator. It allows generating transition probabilities for the given n-gram file.
ModelGenerator. It allows generating n-gram models using different configuration options.
ModelEvaluator. It allows evaluating the performance of n-gram models. 4 metrics are used to compare the performance. These are: Perplexity, accuracy, memory and time taken.
ModelPredictor. It allows predicting the next word, given a set of previous words.
Base. The base class for all the other classes. It allows provides methods for reading and writing files and processing large text files.

Information about the package can be obtained using the command line or the package website. For example, the command: ?wordpredictor returns information about how the given class works and the parameter details for each class method.

Generating the model

The following example shows how to generate a n-gram model.

# The required files
rf <- c("input.txt")
# The test environment is setup
ed <- setup_env(rf, ve)

# The following code generates n-gram model using default options for data
# cleaning and tokenization. See the following section on how to customize these
# options. Note that input.txt is the name of the input data file. It should be
# present in the data directory. ddir is the data directory. mdir is the model
# directory. The output model file, which is def-model.RDS will be placed in
# this directory.

# ModelGenerator class object is created
mg <- ModelGenerator$new(
    name = "def-model",
    desc = "N-gram model generating using default options",
    fn = "def-model.RDS",
    df = "input.txt",
    n = 4,
    ssize = 10,
    dir = ed,
    dc_opts = list(),
    tg_opts = list(),
    ve = ve
)

# Generates n-gram model. The output is the file def-model.RDS
mg$generate_model()

# The test envionment is cleaned up
clean_up(ve)

The above code generates the file def-model.RDS. This file represents the n-gram model.

Predicting words

The following example shows how to predict the next word given a set of words:

# The required files
rf <- c("def-model.RDS")
# The test environment is setup
ed <- setup_env(rf, ve)

# The model file name
mfn <- paste0(ed, "/def-model.RDS")
# An object of class ModelPredictor is created. The mf parameter is the name of
# the model file that was generated in the previous example.
mp <- ModelPredictor$new(mf = mfn)
# Given the words: "how are", the next word is predicted. The top 3 most likely
# next words are returned along with their respective probabilities.
res <- mp$predict_word(words = "how are", 3)

# The test envionment is cleaned up
clean_up(ve)

Analyzing N-grams

The wordpredictor package includes a class called DataAnalyzer, that can be used to get an idea of the frequency distribution of n-grams in a model. The model generation process described above, creates an n-gram file in the model directory.

For each n-gram number less than or equal to the n-gram size of the model, a n-gram file is generated. In the example above the n-gram size of the model is 4. So 4 n-gram files are generated in the model folder. These files are: n1.RDS, n2.RDS, n3.RDS and n4.RDS. The n2.RDS file contains n-grams of size 2.

The following example plots the top 10 most occurring bi-grams along with their frequencies:

# The required files
rf <- c("n2.RDS")
# The test environment is setup
ed <- setup_env(rf, ve)

# The file name
fn <- paste0(ed, "/n2.RDS")
# An object of class DataAnalyzer is created. The fn parameter is the path to
# the n-gram file.
da <- DataAnalyzer$new(fn = fn)
# The top 10 most occurring features are plotted
df <- da$plot_n_gram_stats(opts = list(
  "type" = "top_features",
  "n" = 10,
  "save_to" = NULL,
  "dir" = NULL
))


# The test envionment is cleaned up
clean_up(ve)

The following example plots the n-gram frequency coverage. It shows the percentage of n-grams with frequency 1, 2 … 10.

# The required files
rf <- c("n2.RDS")
# The test environment is setup
ed <- setup_env(rf, ve)

# The file name
fn <- paste0(ed, "/n2.RDS")
# An object of class DataAnalyzer is created. The fn parameter is the path to
# the n-gram file.
da <- DataAnalyzer$new(fn = fn)
# The top 10 most occurring features are plotted
df <- da$plot_n_gram_stats(opts = list(
  "type" = "coverage",
  "n" = 10,
  "save_to" = NULL,
  "dir" = NULL
))


# The test envionment is cleaned up
clean_up(ve)

The following example shows how to get the list of bi-grams starting with “great_” along with their frequencies. It also shows how to get the frequency of the bi-gram “great_deal”.

# The required files
rf <- c("n2.RDS")
# The test environment is setup
ed <- setup_env(rf, ve)

# The file name
fn <- paste0(ed, "/n2.RDS")
# An object of class DataAnalyzer is created. The fn parameter is the path to
# the n-gram file.
da <- DataAnalyzer$new()
# Bi-grams starting with "great_" are returned
df <- da$get_ngrams(fn = fn, c = 10, pre = "^great_*")
# The data frame is sorted by frequency
df <- df[order(df$freq, decreasing = T),]
# The frequency of the bi-gram "great_deal"
f <- as.numeric(df[df$pre == "great_deal", "freq"])

# The test envionment is cleaned up
clean_up(ve)

Customizing the n-gram model

The dc_opts parameter to the ModelGenerator class specifies the data cleaning options. The following code shows the data cleaning options and their default values:


# @field dc_opts The options for the data cleaner object.
#   min_words -> The minimum number of words per sentence.
#   line_count -> The number of lines to read and clean at a time.
#   save_data -> If the combined processed lines should be saved.
#   output_file -> Name of the output file used to store the data.
#   sw_file -> The stop words file path.
#   dict_file -> The dictionary file path.
#   bad_file -> The bad words file path.
#   to_lower -> If the words should be converted to lower case.
#   remove_stop -> If stop words should be removed.
#   remove_punct -> If punctuation symbols should be removed.
#   remove_non_dict -> If non dictionary words should be removed.
#   remove_non_alpha -> If non alphabet symbols should be removed.
#   remove_extra_space -> If leading, trailing and double spaces
#     should be removed.
#   remove_bad -> If bad words should be removed
dc_opts = list(
  "min_words" = 2,
  "line_count" = 1000,
  "save_data" = T,
  "output_file" = NULL,
  "sw_file" = NULL,
  "dict_file" = NULL,
  "bad_file" = NULL,
  "to_lower" = T,
  "remove_stop" = F,
  "remove_punct" = T,
  "remove_non_dict" = T,
  "remove_non_alpha" = T,
  "remove_extra_space" = T,
  "remove_bad" = F
)

The tg_opts parameter to the ModelGenerator class specifies the token generation options. The following code shows the token generation options and their default values:


# @field tg_opts The options for the token generator obj.
#   min_freq -> All ngrams with frequency less than min_freq are
#     ignored.
#   stem_words -> If words should be transformed to their stems.
tg_opts = list(
  "stem_words" = F,
  "min_freq" = -1
)

Evaluating model performance

The wordpredictor package allows evaluating n-gram model performance. It can measure the performance of a single model as well as compare the performance of multiple models. When evaluating the performance of a model, intrinsic and extrinsic evaluation is performed.

Intrinsic evaluation measures the Perplexity score for each sentence in a validation text file. It returns the minimum, maximum and mean Perplexity score for the sentences.

Extrinsic evaluation measures the accuracy score for the sentences in a validation text file. It tries to predict the last word of each sentence. If the word was correctly predicted, then the accuracy count is increased. The extrinsic evaluation returns the number of valid and invalid predictions.

The following example shows how to evaluate the performance of a model:

# The required files
rf <- c("def-model.RDS", "validate.txt")
# The test environment is setup
ed <- setup_env(rf, ve)

# The model file name
mfn <- paste0(ed, "/def-model.RDS")
# The validation file name
vfn <- paste0(ed, "/validate.txt")
# ModelEvaluator class object is created
me <- ModelEvaluator$new(mf = mfn, ve = 2)
# The performance evaluation is performed. The performance stats are returned as
# a data frame and also saved within the model file itself.
stats <- me$evaluate_performance(lc = 20, fn = vfn)

# The test envionment is cleaned up
clean_up(ve)

Demo

A DEMO application demonstrates how to make word predictions. It is based on the Shiny package. It allows predicting the next word based on the given set of words. It displays the 10 most likely words along with their respective probabilities.

The demo app is based on Shiny platform. It consists of two files. server.r and ui.r. The n-gram model file must be present in the same folder as the two files. It can be generated using the ModelGenerator class.

Website

The wordpredictor website provides details about how the packages works. It includes code samples and details of all the classes and methods.

Benefits

The wordpredictor package provides an easy to use framework for working with n-gram models. It allows n-gram model generation, performance evaluation and word prediction.

Limitations

The n-gram language model requires a lot of memory for storing the n-grams. The wordpredictor package has been tested on a machine with dual core processor and 4 GB of RAM. It works well for input data files of size less than 40 Mb and n-gram size 4. For larger data files and n-gram size, more memory and CPU power will be needed.

Future Work

The wordpredictor package may be extended by adding support for different smoothing techniques such as [Good-Turing](https://en.wikipedia.org/wiki/Good%E2%80%93Turing_frequency_estima tion), Katz-Back-off and handling of Out Of Vocabulary Words.

Support for different types of n-gram models such as Skip-Grams and Syntatic n-grams.

The wordpredictor package is used for predicting words. It may be extended to support other use cases such as spelling correction, biological sequence analysis, data compression and more. This will require further performance optimization.

The source code is organized using R6 classes. It is easy to extend. Contributions are welcome !.

Acknowledgments

I was motivated to develop the wordpredictor package after taking the courses in the Data Science Specialization offered by John Hopkins university on Coursera. I would like to thank the course instructors for making the courses interesting and motivating for the students.

R6 class for processing large text files

Tue, 25 May 2021 00:00:00 GMT

Introduction

In this article I will describe a R6 class called TextFileProcessor that I had developed for processing large text files. I had developed the class as part of the wordpredictor package.

The class can be used as a base class by classes that need to process large text files a fixed number of lines at a time.

How the class can be used

The TextFileProcessor class, provides a method called process_file. This method processes a given text file a given number of lines at a time by calling user defined methods. The method takes three arguments. A pre-processing method, a processing method and a post-processing method. These methods need to be defined by the user class.

The pre-processing method is called before the processing of the text file starts. The text file is then read a fixed number of lines at a time. The number of lines to read at a time can be configured.

The lines that are read are passed as argument to the processing method. The processing methods processes the lines and returns them. The processed lines may be written to a output text file or they may be combined and then returned after all the lines in the text file have been processed.

The post-processing method is called after all the lines in the text file have been read and processed.

Benefits of using the class

The main benefit of the TextFileProcessor class is that it makes it easy to create applications that need to process large text files. The programmer has to simply provide a method that processes a given number of lines. Pre-processing and post-processing methods may optionally be defined.

Another benefit of the class is that it supports displaying the progress of processing. This can be controlled using the verbose parameter.

Source code

The following is the source code for the TextFileProcessor class:

#' Base class for text file processing
#'
#' @description
#' Provides basic structure for processing text files.
#'
#' @details
#' It provides pre-processing, processing and post-processing functions, which
#' need to be overridden by derived classes. The pre-processing function is
#' called before reading a file. The process function is called for processing a
#' line. The post processing function is called on the processed data.
#' @export
TextFileProcessor <- R6::R6Class(
    "TextFileProcessor",
    public = list(
        #' @description
        #' It initializes the current object. It is used to set the file name
        #' and verbose options.
        #' @param fn The path to the file to clean.
        #' @param lc The number of lines to read and clean at a time.
        #' @param ve Indicates if progress information should be displayed.
        initialize = function(fn = NULL, lc = 100, ve = 2) {
            # If the given file name is not NULL and is not valid
            if (!is.null(fn) && !file.exists(fn)) {
                  stop("The given file name is not valid")
              }

            # The base class attributes are set
            # The file name is set
            private$fn <- fn
            # The verbose option is set
            private$ve <- ve
            # The line count is set
            private$lc <- lc
            # The processed output is set
            private$p_output <- NULL
        }
    ),
    private = list(
        # @field opts The list of file processing options.
        # * **save_data**. If the combined processed lines should be saved.
        # * **ret_data**. If the data should be returned.
        # * **output_file**. Name of the output file used to store the data.
        opts = list(
            "save_data" = F,
            "ret_data" = F,
            "output_file" = NULL
        ),

        # @field lc The number of lines to read and process at a time.
        lc = 100,

        # @field p_output The output of the processing step
        p_output = NULL,

        # @field fn The name of the text file to process.
        fn = NULL,

        # @field ve Indicates if progress data should be printed.
        ve = 0,

        # @field con The input file connection
        con = NULL,

        # @description
        #' Reads the given file one line at a time. It runs the given
        #' pre-processing function before reading the file. It runs the given
        # line processing function for each line. It optionally saves the
        # output of line processing after reading the file or after processing
        # certain number of lines.
        # @param pre_process The pre-processing function.
        # @param process The function used to process each line.
        # @param post_process The function used to perform post processing.
        # @return The combined processed data
        process_file = function(pre_process, process, post_process) {
            # Pre-processing is done
            pre_process()
            # The file is opened
            private$con <- file(private$fn)
            # The connection is opened for reading
            open(private$con)
            # The lines to be read,
            lines <- c()
            # The loop counter
            c <- 0
            # Indicates that data should not be appended
            is_app <- F
            # The output file name
            of <- private$opts[["output_file"]]
            # All lines are read
            while (TRUE) {
                # The lines are read
                lines <- readLines(private$con,
                    n = private$lc,
                    skipNul = TRUE
                )
                # If all the lines have been read
                if (length(lines) == 0) break
                # The lines are processed
                p_lines <- process(lines)
                # If the processed lines are NULL
                if (is.null(p_lines)) next
                # If the data should be saved
                if (private$opts[["save_data"]]) {
                    # The cleaned data is written to file
                    private$write_file(p_lines, of, is_app)
                    # Debug message
                    private$display_msg(
                        paste(length(p_lines), "lines were written"), 1
                    )
                    # Indicates that data should be appended
                    is_app <- T
                }
                # If the processed data should be returned
                if (private$opts[["ret_data"]]) {
                    # The processed output is merged
                    private$p_output <- c(private$p_output, p_lines)
                }
                # The loop counter is increased by 1
                c <- c + 1
                # Debug message
                private$display_msg(
                    paste(private$lc * c, "lines have been processed"), 1
                )
            }
            # The file connection is closed if it is open
            close(private$con)
            # Post processing is performed
            post_process()
            # If the data should be returned
            if (private$opts[["ret_data"]]) {
                # The processed output is returned
                return(private$p_output)
            }
        },

        # @description
        # Prints the given message depending on verbose settings.
        # @param msg The message to be printed.
        # @param min_debug The minimum debugging level
        display_msg = function(msg, min_debug) {
            # If verbose is >= min_debug , then message is displayed
            if (private$ve >= min_debug) {
                print(msg)
            }
        },

        # @description
        # Performs processing on the data. It should be
        # overriden by a derived class.
        # @param lines The lines to process
        process = function(lines) {

        },

        # @description
        # Performs post-processing on the processed data. It should be
        # overriden by a derived class.
        post_process = function() {

        },

        # @description
        # Performs pre-processing on the processed data. It should be
        # overriden by a derived class.
        pre_process = function() {
            return(NULL)
        }
    )
)

The following is an example of how to use the TextFileProcessor class. The DataCleaner class is an example class that allows cleaning large text files. The class is derived from the TextFileProcessor class. The main method of the class is clean_files.

#' Provides data cleaning functionality
#'
#' @description
#' It provides a memory efficient method for removing unneeded
#' characters from text files. It is suitable for cleaning large text files.
#'
#' @details
#' It provides a method for cleaning text files. It allows removing bad
#' words, stop words, non dictionary words, extra space, punctuation and
#' non-alphabet characters. Allows conversion to lower case. It supports large
#' text files.
DataCleaner <- R6::R6Class(
    "DataCleaner",
    inherit = TextFileProcessor,
    public = list(
        #' @description
        #' It initializes the current object. It is used to set the file name
        #' and verbose options.
        #' @param fn The path to the file to clean.
        #' @param ve Indicates if progress information should be displayed.
        #' @export
        initialize = function(fn = NULL, opts = list(), ve = 0) {            
            # The base class is initialized
            super$initialize(fn, opts[["line_count"]], ve)
            # The save_data option of base class is set
            private$opts[["save_data"]] <- opts[["save_data"]]
            # The output_file option of base class is set
            private$opts[["output_file"]] <- opts[["output_file"]]
        },

        #' @description
        #' It provides an efficient method for cleaning text files.
        #' It removes unneeded characters from the given text file with several
        #' options. It allows removing punctuations, bad words, stop words,
        #' non-alphabetical symbols and non-dictionary words. It reads a certain
        #' number of lines from the given text file. It removes unneeded
        #' characters from the lines and then saves the lines to an output text
        #' file. File cleaning progress is displayed if the verbose option was
        #' set in the class constructor. It is suitable for cleaning large text
        #' files.
        clean_file = function() {
            # The information message
            msg <- paste0("Cleaning the file: ", private$fn)
            # The information message is shown
            private$display_msg(msg, 1)
            # The base class process_file function is called
            private$process_file(
                private$pre_process, private$process,
                private$post_process
            )
            # If the data should not be saved
            if (!private$dc_opts[["save_data"]]) {
                # The processed output is returned
                return(private$p_output)
            }
        }
    ),
    private = list(
        # @description
        # It processes the given lines of text.
        # @param lines The lines of text to clean.
        # @return The processed line is returned.
        process = function(lines) {

            return(lines)
        }
    )
)

The following example shows how the DataCleaner class is used:

dc <- DataCleaner$new(
    "path-to-large-text-file", 
    opts = list(
        "line_count" = 100, 
        "output_file" = "path-to-output-file", 
        "save_data" = T), 
    ve = 2)

Using N-gram models for Word Prediction

Tue, 25 May 2021 00:00:00 GMT

Introduction

N-gram models have applications in many areas such as spell checking, biological sequence analysis, speech recognition, word prediction and more. In this article I will provide an overview of n-gram models. I will first describe n-gram models and how they are used to predict words. Next I will describe the steps involved in developing n-gram models for word prediction.

What are n-grams

N-grams are sequences of items such as letters, words, phonemes, syllables etc. The ‘n’ in n-grams refers to the number of items in the sequence. For example the term “4-gram words” refers to four words that occur in sequence.

The n-gram language model

A statistical language model assigns probabilities to a sequence of words. A n-gram language model is based on the Markov assumption that the probability of a word depends only on the probabilities of the previous words. Smoothing techniques are used to assign probabilities to unknown words.

Two benefits of n-gram models are that they are simple and scalable in the sense that they can be developed using small data sets and then applied to data sets of large sizes.

Maximum likelihood estimation (MLE) is used to determine the probability of words in a n-gram model. The probability of a word in a n-gram model is given by P(w) = count(w)/count(wprev), where w is the word and wprev is set of the previous words. count(w) is the number of times the n-gram occurs. count(wprev) is the number of times the (n-1) gram occurs.

Predicting words using n-gram models

It is possible that a language model does not have the information required to determine the probability of a given word and set of previous words. In this case the language model may refer to lower level n-grams. For example if a 4-gram model does not have data for a given word and previous words, then it may check a 3-gram model. If the 3-gram model also does not have the data for the given word and 2 previous words, then a 2-gram model may be checked. This is known as back-off.

Another options is to calculate the probability of the given word by taking a linear combination of all n-gram models. For example the probability given by each n-gram, n-1-gram, n-2-gram etc models is multiple by constants and added together. This gives a weighted probability for the given word. This method is called interpolation.

Evaluating performance of n-gram models

To evaluate the performance of n-gram models intrinsic and extrinsic evaluation methods may be used. In extrinsic evaluation, the n-gram model is embedded into a particular application. The improvement in the application gives a measure of the model’s performance.

It may not be feasible to embed a n-gram model into an application. An alternative is to measure the performance of the model independent of any application. This is called intrinsic evaluation. This can be carried out using Perplexity. Perplexity is a quantitative measure of how perplexed or confused the language model is when required to predict the probability of a given set of words. A low Perplexity value, implies that the model is less surprised to learn about the given set of words. For n-gram models, Perplexity is equal to the nth root of the inverse of the product of probabilities of each word, given the previous words.

Steps involved in generating n-gram language models

The following steps can be used to generate a n-gram language model from a given input text file:

Data Analysis

The first step is to analyze the given data. This involves determining various text statistics such as number of lines of text, word count, file size, type of textual content etc. This will give an idea of the size of the input data as well as the words and symbols that are not useful. This information will be used in the “Data Cleaning” step.

Data Sampling

The next step is to generate three sample files from the given input data file. These files are the train, validate and test files. The train file is used to generate the n-gram model. The validate file is used to evaluate the model’s performance. The test file is used to run unit tests.

The data in these files needs to be non overlapping. For example the first 80% of the lines in the input data file can be used for the train file, the next 10% for the validation file and the last 10% for the test file. If the train and validation files contain the same data, then the model will predict data it already knows, which can result in overly exaggerated performance scores. This is known as over-fitting.

Data Cleaning

The next step is to remove unwanted symbols and words from the train file that was generated in the previous step. For example vulgar words, foreign words, punctuation symbols, non alphabetical symbols etc may be removed. Once the data has been cleaned, the next step is to generate n-gram tokens from the data.

Tokenization

In this step, the cleaned data file is read. N-grams tokens are extracted from the file and saved along with their frequencies to a new file. For example if the language model has a size of 4, then 4-gram, 3-gram, 2-gram and 1-gram tokens are extracted and saved to separate files along with their frequencies.

For example for the sentence: “today is a great day to go shopping”, the 4-grams are: “today is a great”, “is a great day”, “a great day to”, “great day to go”, “day to go shopping”. The 3-grams are: “today is a”, “is a great”, “a great day”, “great day to”, “day to go”, “to go shopping”. The 2-grams are: “today is”, “is a”, “a great”, “great day”, “day to”, “to go”, “go shopping”. The 1-grams are simply the set of all words.

From the above example, we can see that the higher the n-gram size, the more information or context is provided. Each of the above n-grams are stored along with their respective frequencies, which is simply the number of times, the n-gram occurs in the entire document.

Generating transition probabilities

Each n-gram may be regarded as a set of prefix words followed by a next word. The probability of the next word given the prefix words is the transition probability of the word. It is calculated by dividing the number of times the n-gram occurs by the number of times the prefix words occur. Once the transition probabilities have been generated, they can be used to predict words and calculate Perplexity.

Support for n-grams in R

R is a programming language that is well suited for performing tasks related to Data Science. A R package is software that performs a specific task. The packages tm and quanteda are frameworks for text processing. They provide functions for generating and analyzing n-gram tokens.

The wordpredictor package

I had developed the wordpredictor package as part of the Data Science Capstone course on Coursera. The package allows generating and evaluating n-gram models. It also allows predicting words. A DEMO application shows how the wordpredictor package can be used to predict words.

Expressing numbers as a sum of fives and sevens

Fri, 19 Mar 2021 00:00:00 GMT

Introduction

I recently took the What is a proof course on Coursera and found it very useful. The course provides useful knowledge about mathematical proofs. I found the course very useful and interesting. In this article I will describe one of the problems that were given to the students to solve. The problem is related to the concept of Recursion.

Recursion is one method for proving statements mathematically. It is a powerful concept that has important applications in Computer Science. In this article I will describe how to use the technique of Recursion to prove a statement mathematically. I will describe two such problems.

Problem 1

Is is possible to express any number larger than or equal to 8 as a sum of 3s and 5s. For example 8 = 3 + 5, 9 = 3 + 3 + 3, 10 = 5 + 5, 11 = 8 + 3, 12 = 9 + 3, 13 = 10 + 3 and so on.

If we subtract 3 from a given number, we get another number, if we continue subtracting 3 from the number, we will get a number that is either 8, 9 or 10, which we know can be expressed as a sum of 3s and 5s. The process of subtracting 3 from a number is recursive.

Problem 2

Write a program that takes a number between 24 and 1000 as parameter and returns a list of all combinations of 5s and 7s that when added up gives a result that is equal to the number that was initially passed as argument. Also find the largest number that cannot be expressed as a sum of fives and sevens.

If we can find 5 consecutive numbers that can be expressed as a sum of 5s and 7s, then we can express any number larger than the 5 consecutive numbers as a sum of 5s and 7s. We can do this by subtracting 5 from this number until it is equal to one of the 5 consecutive numbers.

Since we can express the 5 consecutive numbers: 24, 25, 26, 27, 28 as a sum of 5s and 7s, we can express any number larger than 28 as a sum of 5s and 7s. So the largest number that cannot be expressed as a sum of 5s and 7s must be 23.

The following code can be used to express any number between 24 and 1000 as a sum of fives and sevens:


# Main function that is used to express the given number as a sum of fives and sevens
def change(amount):
  
  if amount == 24:
    return [5, 5, 7, 7]
    
  # complete this method
  
  if amount == 25:
    return [5, 5, 5, 5, 5]
  if amount == 26:
    return [7, 7, 7, 5]
  if amount == 27:
    return [5, 5, 5, 5, 7]    
  if amount == 28:
    return [7, 7, 7, 7]    
            
  assert(amount >=24 and amount <=1000)
  
  coins = change(amount-5)
  coins.append(5)
  
  return coins
  
# Express 50 as a sum of fives and sevens
coins = change(50)
# Print the result
print(coins)

Solving N Diagonal Problem

Fri, 19 Mar 2021 00:00:00 GMT

Introduction

I took the What is a proof course on Coursera and found it very useful. The course provides useful knowledge about mathematical proofs. I found the course very useful and interesting. In this article I will describe one of the problems that were given to the students to solve.

In the N Diagonal problem, we have a board containing NxN squares. Each square can have 2 diagonals, but the diagonals of neighboring squares should not intersect. A square can have three possible states. Either empty, a left diagonal or a right diagonal. We need to find out if it is possible to have X number of diagonals.

Solution Overview

We have to go one step at a time, from the bottom left square and from left to right. A square can have 3 possible values. 0, 1 and 2. 0 means empty, 1 means diagonal is from right to left. 2 means diagonal is from left to right.

We can use a list to store the square values in Python. Python does not have built in support for arrays, so we can use a list instead.

First we need to find all possible values that a square can have. Since each square has three possible values, there are many possible combinations of values of a squares neighbors.

We can instead check if there is some combination of neighbor values which makes it impossible for the square to have a value of 1. Similarly we need to check if there is some combination of neighbor values which makes it impossible for the square to have a value of 2.

A square cannot have a value of 1 if the left square has a value of 2 or bottom square has a value of 2 or bottom right square has a value of 1.

A square cannot have a value of 2 if the left square has a value of 1 or bottom square has a value of 1 or bottom left square has a value of 2.

Pseudo Code

Here is the pseudo code:

function SolveNDiagonal(n, diag_count, arr, index):
    if sum of 1s and 2s in list is more than N, then 
        print list
        exit

    For each x in arr starting from index upto NxN
        values = GetPossibleValuesOfSquareAtX(n, arr, x)
    For each val in values, 
        if val is  more than 0, then
            SolveNDiagonal(n, diag_count, arr, index+1)
            
The function is called as follows:

SolveNDiagonal(n, diag_count, arr, 0)

Where n=5, diag_count=16, arr is a Python list initialized with 0s. The last parameter is the starting index, which is 0.

Source code in Python

# Main function
# n is the number of rows or cols in the board
# diag_count is the required number of diagonals
# arr is a list of size NxN
# index is the starting position
# The function prints state of all squares on the board
# Such that the board has diag_count number of diagonals
def solve_n_diagonal(n, diag_count, arr, index):    
 
  if (arr.count(1) + arr.count(2)) == diag_count:
    print("Solved for n = " + str(n) + 
            " and diagonal count = " + str(diag_count))    
    print(arr)
    exit()
              
  for x in range(index, n*n):
    values = get_element_values(n, arr, x)
    for val in values:
      arr[x] = val
      if val > 0:
        solve_n_diagonal(n, diag_count, arr, x+1)
        
# The values of neighboring squares
# We are only concerned with the values of squares on the
# left, bottom, bottom right and bottom left         
def get_neighbor_values(n, arr, x):
  neighbor_values = {"left": -1, "bottom_right": -1, "bottom_left": -1, 
                     "bottom": -1}
  if x % n != 0:
    neighbor_values["left"] = arr[x-1]
  if x >= n and (x % n != n -1):
    neighbor_values["bottom_right"] = arr[x-(n-1)]
  if x >= n and (x % n != 0):
    neighbor_values["bottom_left"] = arr[x-(n+1)]      
  if x >= n:   
     neighbor_values["bottom"] = arr[x-n]      
     
  return neighbor_values
  
# Get the possible values that the square at position x can have
# The possible values are returned in a list
def get_element_values(n, arr, x):  
  neighbor_values = get_neighbor_values(n, arr, x)
  
  values = list()
  
  is_one_valid = True
  is_two_valid = True
  
  if (neighbor_values["left"] == 2 or neighbor_values["bottom"] == 2
    or neighbor_values["bottom_right"] == 1):
    is_one_valid = False
  if (neighbor_values["left"] == 1 or neighbor_values["bottom"] == 1
    or neighbor_values["bottom_left"] == 2):
    is_two_valid = False  
    
  if is_one_valid:
    values.append(1)
  if is_two_valid:
    values.append(2)
    
  values.append(0)
  
  return values
  
# Initialize a list of size n*n with 0s  
def initialize_array(n):
  arr = list()
  for x in range(n*n):
    arr.append(-1)
  
  return arr
     
# The number of rows or cols in the board      
n = 5
# The required number of diagonals
diag_count = 16
# The list is initialized
arr = initialize_array(n)
# The main function is called
solve_n_diagonal(n, diag_count, arr, 0)

The above code prints the state of each square on a board, such that a square is either empty or it has a diagonal which does not intersect with the diagonals of its neighboring squares.

Containerization - Quick Tips

Sun, 16 Aug 2020 00:00:00 GMT

Following are some quick tips related to Docker and LXC/LXD containers:

Removing unused Docker images

To remove all unused docker images with no associated container enter the following command:

docker image prune --all

Getting console output of Docker container

To get the console output of a Docker container, enter the following command:

docker logs [container-id]

To get the console output in real time use the -f switch

Running Docker container in detached mode

To run a Docker container in detached mode, use the -d switch. For example:

docker run -d [image-name]

Running a Docker container on server boot

The Docker restart policy determines how a Docker container should be restarted once it exits. The restart policy is specified using the —restart flag. The default value of this flag is “no”. To automatically start the container each time the Docker daemon starts, the restart flag should be set to “always”. For example:

docker run --restart="always" [image-name]

This can be used to automatically run Docker containers when the server boots.

Listing LXC containers

To list LXC containers, the lxc-ls command should be used. It supports flags for listing running and stopped containers. For example to list all running LXC containers, use the following command:

lxc-ls --running

Installing Jahia Community SDK on Debian 9

Sat, 15 Aug 2020 00:00:00 GMT

Introduction

Jahia is a software company which provides enterprise products and services. Its main products are jContent (a content management system), jExperience (combines customer data with customer management for personalized customer experience and jCustomer (an open source customer data platform).

These products are combined in innovative ways to provide solutions such as CMS (Customer Management System), PWA (Headless Progressive Web Applications), Customer Portals and more. Jahia provides a 30 day trial period of most of its products and solutions.

Jahia is supported by an active open source community. It initiated the Apache Unomi project and is an active contributor to the Karaf and Cellar Apache projects, as well as GraphQL Java. Jahia has hundreds of open source repositories on Git Hub.

The Jahia documentation site provides useful documentation for end users, system administrators and developers.

The Jahia module store provides modules that extend Jahia product features.

Trying out Jahia

From the Jahia website we can download Jahia (currently at version 8) after creating a user account. The Downloads section of the website allows downloading Jahia Enterprise and Jahia Community.

The Jahia Enterprise is available for a 30 day trial period. The community edition contains a preinstalled Jahia instance, a demo website and Jahia Studio, which allows managing Jahia modules, website content and more.

To try out Jahia, we can install it as a Docker image using the following commands:

docker pull jahia/jahia-dev:8.0.0.0
docker run -p 8080:8080 jahia/jahia-dev:8.0.0.0

This provides an Enterprise edition of Jahia for a 30 day period. The Jahia instance can be accessed over port 8080.

The community edition has the following features: Content editing, Digital asset management, Multi-language management, Search, Digital portal building, Engagement, Web marketing, Ouf-of-the-box mobile readiness and advanced multi-channels features and Developer tools which includes Jahia Studio and Integrated Development Tools.

Jahia system requirements

The Jahia system requirements are quite demanding. Jahia requires Java JDK version 8 or 11. It requires 4 Gb of RAM and a dual core processor for a development environment and 8 Gb RAM and quad core processor for a production environment.

Java 8 is the default Java version for Debian 9. It may be installed using the command: apt-get install default-jdk.

To install JDK version 11, we need to add the backports repository to the apt sources list using the command:

echo 'deb http://ftp.debian.org/debian stretch-backports main' | sudo tee /etc/apt/sources.list.d/stretch-backports.list

We can then install the JDK 11 using the command: apt-get install openjdk-11-jdk. To see which version of Java was installed, enter the command: java -version. The article Install Java on Debian 9 Operating System describes how to install Java on Debian 9. Java JDK comes in two flavors called OpenJdk and OracleJdk. Installation of both flavors is described in the article.

Also make sure to set the JAVA_HOME environment variable to the path of the JDK installation. On Debian 9 the location of the JDK is /usr/lib/jvm/java-11-openjdk-amd64. The variable should be set in /home/[user]/.bashrc file, where [user] is the system user account used to run Jahia. For example add following to /home/user/.bashrc file:

export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"
export PATH=$JAVA_HOME/bin:$PATH

Installing Jahia community

To install Jahia community, we need to first create an account on the Jahia website and then download the community edition zip file. Once the file has been downloaded, it needs to be unzipped. Next we need to run the file start.sh. The command should be run as a non root user. After that Jahia should be up and running. It should be accessible on port 8080.

Jahia features

I downloaded and installed Jahia first as a Docker image. This provided an out of the box preinstalled Jahia enterprise edition. However it had a 30 day trial period. Next I downloaded the Jahia community edition zip file.

Both Jahia instances are basically a Content Management System (CMS), which allow managing website content from a back-end area. Modules can be used to add features to the website. These modules provide a wide range of features that can make your website function as a simple CMS, or a CRM (customer relationship management).

Jahia has a front-end and a back-end. The front-end is simply the front-end of the default website that was created from the Jahia back-end. The Jahia back-end is also called Jahia Studio. It is accessible over port 8080 on the url /start. For example http://localhost:8080/start. Jahia Studio allows managing modules, websites, users and groups and more.

The Jahia module store provide a wide range of features. For example there are modules related to Business and Commerce, Collaboration and Document Management, Content Types, Marketing, Portals, Social Networks and User Generated content, Templating and Extensions and Tools and Utilities.

Depending on the modules installed, your Jahia instance can function as an Ecommerce site, a Blog, as a Social Network or any other type of site that can be created from Jahia modules. The Jahia documentation on managing modules describes how to create, install and update Jahia modules.

Conclusion

Jahia is an excellent enterprise content management system. It is very flexible and has a lot of features. It is easy to install and is well documented.

Adding Table of Contents to a Gatsby website

Sun, 26 Apr 2020 00:00:00 GMT

Introduction

Gatsby is a modern CMS that is well suited for publishing Blog posts. However managing a Gatsby Blog is similar to managing a web application and can be difficult for those not familiar with Software Development.

One of the useful features of Gatsby is that it provides a clean separation between the different stages involved in Web Development. For example connecting the blog to a data source, adding features using plugins etc. Features such as Sitemaps, Search Engine Optimization (SEO), Pagination etc can be easily added by installing and configuring plugins. In an earlier post I described my experience with migrating a blog to Gatsby.

Readers of Blog posts can sometimes lose the overall picture when reading long articles. A table of contents describes the main components of an article and gives an overall picture. In this blog post I will describe my experience of adding table of contents to a Gatsby Blog.

Table of contents script

Gatsby provides a plugin called Remark Table of Contents, which allows adding a table of contents to markdown files. To use the plugin, We would have to add table of contents markdown to all post files. This would take a long time on large blogs. Also if I wanted to update the table of contents layout or position, then I would have to edit all markdown files.

The Gatsby website is based on the Lumen Starter Theme. The theme fetches data from Markdown files using GraphQL. It converts the Markdown to HTML and displays the HTML inside React Js template files.

The blog was initially based on a custom PHP framework called Pak Php Framework. I had created a script in PHP language for generating table of contents from HTML text. I converted this script to JavaScript and used it to generate the table of contents for my new Blog.

I managed to convert the script and published it as a NPM module. The NPM package is called Node.js Table of Contents Generator. The script can be used to add table of contents to a Gastby website. It can also be used on other headless CMSs.

The script takes as input article text in HTML format and produces an unordered nested HTML list, which is the table of contents. The script also returns updated article text, which includes heading ids.

Adding Table of Contents to Gatsby blog

To add Table of Contents, you need to follow these steps:

Install the table of contents script using the command:

npm i @pakjiddat/toc --save

Create a new Gatsby component called Toc, under the Post component. To create the component, create a folder called Toc under the src/components/Post folder. Add a file to this folder called Toc.tsx with following contents:

import React from 'react'
import * as tocStyles from "./Toc.module.scss";

class Toc extends React.Component {
  render() {

    const tocList = this.props.tocList;
    const visibilityClass = this.props.visibilityClass
    
    return (
      <div className={tocStyles.tocList + " " + visibilityClass} id="toc-box">
        <h5 className={tocStyles.tocHeader}>Table Of Contents</h5>
        <div dangerouslySetInnerHTML={{__html: tocList}} />
      </div>
    )
  }
}

export default Toc;

Next add a file called Toc.module.scss in the same folder, with following contents:

.tocList {
  top: 15%;
  left: 76%;
  margin-right: 30px;
  position: fixed;
  border: 2px solid gray;
  border-radius: 2px;
  background-color: white;
  max-height: 350px;
  overflow: auto;
  width: 20%;
}

.tocHeader {
  padding: 0.75rem 1.25rem;
  margin-bottom: 0;
  margin-top: 0;
  background-color: rgba(0,0,0,.03);
  border-bottom: 1px solid rgba(0,0,0,.125);
}

.visible {
  display: block;
  
}

.hidden {
  display: none;
}

ul {
  margin-bottom: 0 !important;
}

li {
  margin-bottom: 0 !important;
}

.sidebarBtns {
  display: none;
  position: fixed;
  bottom: 50px;
  right: 10%;
  z-index: 99;
  font-size: 1.125rem;
  border: none;
  outline: none;
  background-color: transparent;
  color: white;
  cursor: pointer;
  padding: 0.9375rem;
  border-radius: 0.25rem;
  background-image: url("/arrow-up.svg");
  background-repeat: no-repeat;
  width: 50px;
  height: 50px;
  background-position: center;
}

.toggleTocBtn {
  right: 10%;
  background-image: url("/toggle-toc.svg");
  background-color: cadetblue;
}

@media screen and (max-width: 1200px) {
  .tocList {
    position: relative;
    left: 20%;
    margin-top: 5%;
    width: 30%;
  }
}

The Toc.module.scss file is used to style the table of contents. Update this file if you want to change the look and feel of the table of contents.

Next add a file called index.ts in the same folder, with following contents:

export { default as Toc } from "./Toc";

Next import this component into your blog post template file. This can be done using the following code:

import { Toc } from "./Toc";
const toc    = require("@pakjiddat/toc/index");
import * as tocStyles from "./Toc/Toc.module.scss";

Next within your blog post template file, generate the table of contents from your article HTML text using:

const tocData = toc.Generate(post.html);
  let page_html = tocData.updatedText;
  let toc_list = tocData.tocList
  let visibility_class = tocStyles.visible
    
  if (tocData.updatedText === "") {
    page_html = post.html
    toc_list = ""
    visibility_class = "hidden"
  }

The above code assumes that your article HTML text is in the post.html variable

Next display the updated article text returned by the table of contents script. The updated article text is in the variable: page_html. Use this variable instead of the post.html variable.
Add the table of contents component to your post template file using the following code:

<Toc tocList={toc_list} visibilityClass={visibility_class} />

After following the above steps you should have a table of contents on each blog post.

Limitations

The table of contents script requires that your article headings are ordered correctly. i.e lower order headings are nested within higher order headings.

Future work

The table of contents script may be extended so it adds an anchor icon to each article heading. The script may also be extended to support extraction of headings from Markdown files.

Conclusion

In this blog post I have described how to add table of contents to a Gatsby Blog. The main steps outlined in the article may be applied to other headless CMSs.

Adding scroll to top button to a Gatsby website

Sun, 26 Apr 2020 00:00:00 GMT

Introduction

Scrolling to top is a useful feature for Blogs. It allows readers to scroll to the top of the page any time with a single click of a button. In this article I will describe how to add a scroll to top button to a Gatsby website.

Client side script

To add scroll to top feature, we need to add a button and a JavaScript file. The JavaScript file will be used client side. Create a file called gatsby-browser.ts in the root folder. It should have following contents:

import "./src/assets/scss/main.scss";
export { wrapRootElement } from "./internal/gatsby/wrap-root-element";

/** The SideBarBtns class */
class SideBarBtns {

  /** Used to register the scroll event handler */
  initialize() {
    /** When the user scrolls down 300px from the top of the document, show the buttons */
    window.addEventListener("scroll", this.toggleButtons)
    /** Event handler for toogle toc button */
    document.getElementById("toggle-toc-btn").addEventListener("click", this.toggleTocBox);
    /** Event handler for scroll to top button */
    document.getElementById("scroll-btn").addEventListener("click", this.scrollToTop);
  }

  /** Displays/Hides the buttons */
  toggleButtons() {
    /** If the scroll top button does not exist, then function returns */
    if (!document.getElementById("scroll-btn")) return;
    /** If the current current scroll is 300px or more */
    if (document.body.scrollTop > 300 || document.documentElement.scrollTop > 300) {
      /** The scroll to top button is displayed */
      document.getElementById("scroll-btn").style.display = "block";
      /** The toggle toc button is displayed */
      document.getElementById("toggle-toc-btn").style.display = "block";
    } else {
      /** The scroll to top button is hidden */
      document.getElementById("scroll-btn").style.display = "none";
      /** The toggle toc button is hidden */
      document.getElementById("toggle-toc-btn").style.display = "none";
    }
  }

  /** When the user clicks on the button, scroll to the top of the document */
  scrollToTop() {
    /** The user is scrolled to the top of the page */
    document.body.scrollTop = 0;
    document.documentElement.scrollTop = 0;
  }

  /** When the user clicks on the button, make the toc box invisible */
  toggleTocBox() {
    /** If the toc box is hidden */
    if (document.getElementById("toc-box").style.display === "none") {
      /** The toc box is displayed */
      document.getElementById("toc-box").style.display = "block";
    } else {
      /** The toc box is displayed */
      document.getElementById("toc-box").style.display = "none";
    }
  }
}

window.addEventListener("load", () => {
  setTimeout(() => {  
      /** The SideBarBtns object is created */
      let sidebarbtns = new SideBarBtns();
      /** If the current page is an article page */
      if (document.getElementById("scroll-btn")) {
        /** The SideBarBtns object is initialized */
        sidebarbtns.initialize();
      }
  }, 1000);
})

window.navigation.addEventListener("navigate", (event) => {
    setTimeout(() => {  
      /** The SideBarBtns object is created */
      let sidebarbtns = new SideBarBtns();
      /** If the current page is an article page */
      if (document.getElementById("scroll-btn")) {
        /** The SideBarBtns object is initialized */
        sidebarbtns.initialize();
      }
  }, 1000);
})

The above file will be automatically included by Gatsby.

Adding the scroll to top button

Next add the following scroll to top button to your post template file:

<button id="scroll-btn" className={styles.sidebarBtns + " " + styles.scrollBtn} title="Scroll to top" aria-label="Scroll to top"></button>

Styling the button

Add the following CSS styles to the local Post.module.scss file of your post template component.

.sidebarBtns {
  display: none;
  position: fixed;
  bottom: 50px;
  right: 10%;
  z-index: 99;
  font-size: 1.125rem;
  border: none;
  outline: none;
  background-color: transparent;
  color: white;
  cursor: pointer;
  padding: 0.9375rem;
  border-radius: 0.25rem;
  background-image: url("/arrow-up.svg");
  background-repeat: no-repeat;
  width: 50px;
  height: 50px;
  background-position: center;
}

.scrollBtn {
  right: 15%;
  background-image: url("/arrow-up.svg");
  background-color: burlywood;
}

The arrow-up icon which is used as the button background can be downloaded from:

https://pakjiddat.netlify.com/arrow-up.svg

The icon should be placed in a folder called static in the root folder.

Conclusion

In this blog post I have described how to add scroll to top button to a Gatsby blog. The button’s style can be changed by editing the given CSS code.

Node.js Table of Contents Generator

Sun, 26 Apr 2020 00:00:00 GMT

Introduction

The Node.js Table of Contents Generator is a Node.js module for generating table of contents from HTML text.

Installation

npm i @pakjiddat/toc --save

The above command will download the package to the node_modules folder. It will also add an entry to your package.json file

Usage

const toc    = require("@pakjiddat/toc/index");

var tocData  = toc.Generate(data);

In the above command the data parameter is the article text in HTML format. The tocData variable contains the following:

tocList. The table of contents list. It is formatted as a nested unordered HTML list
updatedText. The updated article text with id attribute added to all the headings
headingCount. The number of headings found in the article
errorMsg. An error message describing the error

Structure

The only required file is index.js in the root project folder. It contains a single exported class called TOC. The class has one public function called Generate which generates table of contents for the given article text.

The article text must be in HTML format. It returns the table of contents formatted as a nested unordered HTML list. Each item in the nested list links to a heading in the main article.

How it works

The TOC class extracts all HTML tags from h1 to h6. It then creates an unordered HTML list in the same order as the article headings. It also adds an id attribute to each h tag. This allows the items in the table of contents to link to the headings in the article.

The TOC class uses a divide and conquer approach based on recursion for extracting the headings from the article text. It first extracts all the h1 tags from the article text. It then extracts the text between each successive h1 tag and then extracts the h2 text from this text. It then extracts the text between each successive h2 tag. In this way it continues until all tags from h1 to h6 have been extracted. If a tag for example h1 is not found, then the next higher order tag which is h2 is extracted.

Limitations

The limitations of the package are:

It only supports article text in HTML format
It requires that article headings are arranged in numeric order. i.e high order headings contain lower order headings. For example h1 tags are followed by h2-h6 tags

Testing

To test the package, extract the file tests/data.zip. This file contains about 150 articles with headings. Copy all articles to the tests/data folder. Next run the command:

npm test

This will run the test code in tests/test.js. The test code reads each file in the data folder and extracts the headings from the file. It then checks if the headings have been extracted. In case of errors an error is message is logged to the console.

Evolution of web development

Mon, 13 Apr 2020 00:00:00 GMT

Introduction

Websites may be developed in different ways. We can code a website from scratch or we can use libraries and frameworks. Or a combination of both approaches. All websites have a front-end end, which is displayed in a web browser to the end user. The website front-end may fetch data from a server. The data provided by the server may reside in flat files or in databases. The server may return raw data or it may process the data and combine it with presentation code.

There are numerous web development frameworks that use different approaches, but the end result is the same. It is code that runs in a web browser, which may or may not access a web server after the initial load. Thats how websites used to work. A web developer was concerned only with developing websites for browsers.

During last few years, website usage has become very common and the role of the web developer has consequently evolved. As the demand for websites grew, there was a need for websites that work across different browsers and operating systems. Technologies such as response website design and poly-fills were introduced to meet this demand. Responsive website design is a collection of design practices that allows a website to look the same, irrespective of the browser’s view-port size. Poly-fills provide modern features to older browsers that do not support these features.

Cross platform development

More recently there has been a demand for websites that can be accessed from mobiles applications and desktops. The web developer is no longer making websites only for the browser. Businesses want their customers to be able to access their websites from any browser and device. This demand has resulted in technologies such as Electron, Cordova and Capacitor which allow mobile applications to be developed using the familiar HTML, JavaScript and CSS technology stack. Frameworks such as the Ionic framework, allow websites to run on mobile devices, desktops and browsers using a single code base.

The HTML, JavaScript and CSS stack

HTML, JavaScript and CSS are the basic technologies that power websites. Over the years there has been a lot of changes to these technologies. For instance the HTML standard has had different versions such as HTML 4, XHTML and now HTML 5. Similarly CSS has had two major versions. CSS 2 and now CSS 3.

JavaScript has also had different versions based on the ECMAScript standards. The ECMAScript is a set of specifications for a scripting language. JavaScript implements the ECMAScript standards. ECMAScript has been revised over the years. The most recent version is called ECMAScript 2019.

Extension of JavaScript and the ECMAScript standard

JavaScript was initially designed to run only in web browsers. But now it has been extended and modified to run on servers as well. Frameworks such as Dart, CoffeeScript, Flutter, TypeScript and Node.js are all based on JavaScript and ECMAScript standards. They provide new capabilities to the JavaScript language such as static typing and server side rendering.

Previously JavaScript code only ran on the client side within a web browser. Now it can run on the server side as well. Frameworks such as React and Angular, which are based on Node.js involve JavaScript execution on server side as well as client side. These frameworks are becoming very popular and are commonly being used by web developers these days. These frameworks can be used to write applications that can be deployed across different platforms. For example an application based on React framework can be run as a Hybrid mobile application using the React Native framework or as a desktop application using the Electron framework. The application may be accessed as a normal website from a web browser or as a Progress Web Applications (PWA).

A Web Developer who is experienced with HTML, JavaScript and CSS should not have problems with learning the new frameworks such as React, Angular and Vue. The main difference between these frameworks and the pure JavaScript approach is that these frameworks use a declarative approach towards programming as apposed to the commonly used procedural approach.

Declarative vs Procedural approach

In the declarative approach the programmer is simply concerned with flow of the application. He is not concerned with the low level technical details. In procedural programming the programmer is concerned with the low level details. Procedural programming has limitations and may not be suitable in certain cases. For example development of a website that needs to work across various devices and browser sizes, is not feasible using the Procedural approach. The developer has to be concerned with too many low level details.

Conclusion

The role of web developers is becoming more and more specialized and focused. New frameworks and tools are being introduced for accomplishing work efficiently. Basic knowledge about the new tools and technologies can be very useful. For example having a good knowledge of a framework such as Gatsby can allow one to develop websites efficiently and according to customer requirements.

Previously a web developer was only concerned with making websites following a simple workflow. Web Development has changed and now requires knowledge of many different subjects. A Web Develop may benefit from knowledge of Software Engineering methodologies such as Agile. He may need to add features to a website that requires knowledge about machine learning and artificial intelligence. He may also benefit from knowledge outside the domain of computer science that is relevant to the website.

Web Development is an exciting occupation that can provide opportunity to learn new things. The main requirement for becoming a successful web developer in my opinion is to keep learning and refreshing ones knowledge about areas that are related to ones work.

In my opinion, it is important for a web developer to keep him self updated about the latest trends in web development. Instead of focusing only on the day to day businesses requirements, he should learn about the new trends in web development. Learning new things keeps the mind healthy and productive and helps in all areas of life.

Migrating blog to Gatsby

Sun, 12 Apr 2020 00:00:00 GMT

Introduction

Gatsby is a modern framework for developing websites. It requires knowledge of the React JavaScript framework and Node.js. Gatsby uses an innovative approach toward developing websites which is quite different from the common website development workflow.

The main benefit of using Gatsby is that is provides a clean separation of all the steps involved in web site development. Each step in the web development cycle is well documented and easy to implement. For example website data is fetched using the GraphQL query language. The website front-end is implemented using React components. Website optimization is accomplished using web pack. Gatsby also provides a continuous integration workflow and supports static site hosting services such as Netlify. Netlify has a free plan which allows free hosting of static websites.

Gatsby websites are developed using React framework. Once a website has been developed, it is deployed as static HTML pages. The websites pages are loaded dynamically using JavaScript.

I decided to migrate my web development blog to Gatsby mainly because of the learning experience and also to save cost. My blog was based on a custom made PHP Framework. The blog worked well, but was difficult to update and also was expensive to host.

It seemed to me there is no reason to pay for hosting a blog, when you can host one for free. I decided to switch to Gatsby because it provides an innovative approach towards web development and also because it produces static websites, which can be hosted for free.

Gatsby provides a component based, declarative approach towards website development, which is a refreshing change from the procedural approach commonly used for developing websites. Usually when developing websites, we need to be concerned with the low level details of each feature. With Gatsby, we can add features easily by simply installing and configuring pre-built components.

Installation

The Gatsby Quick Start guide describes how to install Gatsby. Gatsby is installed as a Node.js package from the command line using:

npm install -g gatsby-cli

To create a new site, enter the command:

gatsby new gatsby-site

We may specify a starter template in the above command. Gatsby has hundreds of starter templates that can be used as a starting point for creating a website. Next start a development server by entering the command:

gatsby develop

This will start a development server on port 8000. You should be able to access the website on: http://localhost:8000. If you edit any files in the src/pages folder, It will cause the website to automatically reload in the browser and reflect the changes made to the files. Once the website has been developed it can be deployed as static HTML files using the following commands:

gatsby build
gatsby serve

The first command will generate static HTML files, while the second command will start a web server for serving the files. The Gatsby Tutorial page provides useful tutorials for getting started with Gatsby web development.

Gatsby plugins

Gatsby has hundreds of plugins that can be used to add features to your website. For example there are plugins for adding sitemap functionality, table of contents, SEO, Authentication, data loading and transformation, Pagination, Forms, RSS Feeds, Comments and more. Gatsby plugins are installed as NPM modules from the command line. Each plugin has its own configuration details.

Loading data

Data can be loaded to a Gatsby site by using the Gatsby CreatePages API or by using GraphQL. Small sites should use the CreatePages API, while large sites should consider using GraphQL. GraphQL was developed by Facebook and is the most common method used for loading data into React components.

Gatsby can load data from different sources, such as MarkDown files, WordPress, Headless CMSs and more.

Website Deployment

Gatsby provides support for deploying to different static site hosting providers. The Gatsby Deployment and hosting guide lists all the hosting providers supported by Gatsby.

To deploy a Gatsby site to Netlify, we need to first commit the website to a Git Hub repository. Next we need to sign-up for a Netlify account. After that we need to create a new site in Netlify and connect the site with our Gatsby repository. The site should have a unique name.

Once the repository is connected to our Netlify site, the site is built by Netlify and then deployed to a Content Delivery Network (CDN). The website should now be accessible on the URL: https://your-site.netlify.com.

Netlify automatically provides HTTPS for the website. Every time we make a commit to the Git Hub repository, Netlify builds the Gatsby site and deploys the static content to its Content Delivery Network (CDN). This provides a continuous integration approach for developing the website.

Conclusion

In my opinion Gatsby is an excellent option for hosting blogs. It is well suited for use by web developers. It provides an innovative approach to website development which is quite different from that used by frameworks such as LAMP (Linux, Apache, MySQL, PHP) and ASP.net. It cleanly separates the main web development tasks. It is well documented and has an active open source community.

Separating source code and development environment using Docker

Mon, 30 Mar 2020 00:00:00 GMT

Introduction

A common problem faced by developers involves setting up development environments. Tools such as Docker and Vagrant allow separating source code and development environments.

These tools can be used in different ways leading to different possible workflows for developers. In this article I will describe a useful workflow based on Docker. The main benefit of this workflow is that it allows separating source code and development environment.

Concept

The main idea behind the workflow is to separate the application source code from the development environment. The source code is kept on the host computer and mounted in the Docker container. The Docker container runs the development environment.

Currently 3 development environments are supported. LAMP (for creating applications based on LAMP stack), NodeJs (for creating applications based on Node.js) and R-Env (for creating applications based on R Programming language)

Each development environment has its own Docker image. For example to develop Nodejs applications, you need to use the Nodejs Docker image. To develop LAMP based applications, you need to use the LAMP docker image.

The workflow for each development environment is the same. You need to first download a Docker image, then download and extract the base directory structure for the development environment. The base directory structure is provided as a zip file. See the following guides on how to develop applications for each development environment:

Usage

Conclusions

Docker is a useful tool that simplifies developer workflow. It allows developers to focus on developing applications instead of managing the development environment.

Pak Php CRM

Sat, 28 Mar 2020 00:00:00 GMT

Introduction

Pak Php CRM is a simple web based task management application. It can be used by web developers to keep track of tasks, income, billing, goals and more.

Pak Php CRM is written in PHP language and uses MySQL for storing data. It is based on the Pak Php Framework.

Features

The main idea behind the Pak PHP CRM is to provide a simple application for managing basic developer tasks such as task management, time tracking, project management etc. The application can be easily customized. The idea is to allow developers to create simple applications quickly using form and table widgets.

The Pak PHP CRM allows time tracking, management of tasks, goals, income, billing, loans, products and services, customers and projects
It allows user management
It can easily be extended with more features
The website front-end is based on WC.CSS
The website back-end is written in PHP
The website is mobile friendly
The website has good browser response time

Requirements

The Pak Php CRM is based on PHP and MySQL. Installing the website on your own server requires Php 7.2 and above. The website source code is based on the Pak Php Framework. The source code is fully commented and compliant with the PSR-2 coding guidelines. The source code for the website is modular and easy to extend with new features.

Installation

The following steps can be used to install the Pak Php CRM application on your own server:

Download the source code from GitHub
Move the source code to the document root of a virtual host
Create a database and import the contents of the file: crm/data/pak-php-crm.sql to the database. Note down the credentials used for connecting to the database
Enter the database credentials in the file crm/config/RequiredObjects.php
In the file: crm/Config.php, on line 41 enter the domain names that will be used to access the website
Customize the following variables in the file: crm/config/General.php. $config[‘app_name’], $config[‘dev_mode’] and $config[‘site_url’]
Set the $config[‘pear_folder_path’] variable in the file: crm/config/Path.php. The variable should be set to the path of the pear installation.
Visit the website in a browser

Installing Ghost on Debian Stretch

Fri, 24 Jan 2020 00:00:00 GMT

Introduction

Ghost is an open source blogging platform similar to WordPress. It is based on a modern web architecture and has very good performance and security. It is very popular and is used by major corporations including Mozilla, NASA, Apple and more.

Ghost Architecture

Ghost is based on the Node.js platform. It is completely written in JavaScript. It is easy to install and use. According to the Ghost website, the performance of Ghost is 19 times that of WordPress.

The Jamstack

Ghost is based on an modern architecture for websites known as Jamstack. The JAM stands for JavaScript, APIs and Markdown. A Jamstack is essentially a static website which uses API calls to load content.

Static site generators such as Jekyll, Next, Nuxt and Gatsby are used to develop Jamstack sites. A Jamstack site consists of plain html pages containing CSS and JavaScript only. The HTML pages can be deployed to a content delivery network (CDN) and delivered to browsers efficiently and with minimum latency.

Jamstack sites do not require a web server. They usually load much faster and are more secure than websites with a backend such as WordPress and Drupal. Jamstack sites provide a better developer experience since they decouple features. They are also cheaper to scale.

Ghost components

Ghost consists of 3 main components. A Node.js based API, an admin client based on Ember.js and a frontend theme layer based on Handlebars.js. Ghost is also a headless CMS, which means the backend and frontend are decoupled and can be used independantly of each other. Content published using the Ghost admin can be displayed on a frontend site created using a framework such as Next, Nuxt or Gatsby. The frontend site contains only HTML pages and is termed as the Jamstack site.

Ghost storage

Ghost uses the Bookshelf.js ORM for accessing the data. This allows a wide range of databases to be used with Ghost. Sqlite3 and MySQL databases are officially supported and recommended for development and production use respectively. Ghost uses the local file system for storing images. Other storage types are supported such as S3, Content Delivery Networks (CDNs), databases, Google Cloud Storage and more.

Installing and running Ghost

Ghost has several installation options. Ghost Pro is a fully managed platform as a service which allows your Ghost site to be managed and run on the cloud. Ghost may also be installed locally from source for development, deployed within a Docker container or installed locally as a site.

A local installation of Ghost requires a recent version of Node.js. Node versions 8, 10 and 12 are supported. Ghost requires about 1 GB of RAM. Ubuntu Operating system versions 16.04 and 18.04 are officially supported.

To install Ghost as a local site run the command: npm install ghost-cli@latest -g. This will install the Ghost command line interface tool as a global NPM application. To create a new site, simply change to the installation directory of your choice and enter the command: ghost install local. This will create a new installation of a Ghost site.

Next enter the command: ghost start. This will start a web server on port 2368. You should be able to access the Ghost site on: http://localhost:2368. The Ghost Admin can be accessed on: http://localhost:2368/ghost. To stop the Ghost server enter the command: ghost stop from the folder containing the Ghost site.

To run your Ghost site automatically on server boot, install the forever and forever-service NPM modules globally using the commands:

npm install -g forever
npm install -g forever-service

Next install the Ghost service using the command: forever-service install [service-name] —script versions/[version-name]/index.js. The command should be issued from the site root folder. Replace [service-name] with the name of the service such as Ghost. Replace [version-name] with the current version of the Ghost CMS. Your Ghost site will now start automatically on server boot. Also you can manage your Ghost site as a system service. For example you can stop the site using the command: service ghost stop. You can start the site using the command: service ghost start.

Configuring Ghost

Ghost can be configured by simply editing configuration files. By default Ghost is installed in development mode with SQLite3 database. To switch to production mode set an environment variable using the command: NODE_ENV=production node index.js. The configuration file for development mode is config.development.json, while the configuration file for production mode is: config.production.json. The configuration files are in JSON format. The main configuration options are URL, database and mail. The Ghost configuration guide describes the different Ghost configuration options.

How to use Ghost

Ghost workflow

The workflow for using Ghost as a headless CMS is very simple. The user authors the content using the Ghost admin client. The content is then provided to a frontend framework such as Next, Nuxt or Gatsby using the Ghost API. The frontend framework then generates the static HTML pages.

The Ghost admin

The Ghost admin is a simple frontend, which uses the Ghost Admin API for saving and retrieving content. The Ghost admin can be accessed from the URL: http://your-domain/ghost. The Ghost admin allows creating posts and pages. It also allows creating tags and users.

The settings area allows installing the front-end theme. The theme can be installed by uploading a zip file.The Ghost admin allows managing the navigation menu items. It also allows importing and exporting content and deleting all content

The Ghost API

Ghost provides a content API as well as an admin API. The content API allows front-end frameworks to access content. The admin API allows content authoring applications to post content. This allows publishers to use their favorite authoring application, without being concerned with the front-end. The Ghost API decouples the front-end and admin features and allows Ghost to work as a Headless CMS.

Ghost themes

Ghost themes are based on the Handlebars template language, which provides a separation between the HTML and the dynamic JavaScript. The Handlebars language uses helpers to insert dynamic content. Quality free and paid Ghost themes can be purchased from the Ghost marketplace.

Ghost features

Ghost provides mobile and desktop apps for authoring content. It also has built in support for SEO, Accelerated Mobile Pages (AMP), Detailed Structured data and Subscription by RSS, Email and Slack.

Ghost has hundreds of integrations with popular tools and services, including Discourse, Slack, Google Analytics, Zapier and more.

Ghost has a built in membership and subscription feature. It allows website owners to generate revenue by providing paid membership and subscription services. The membership and subscription feature is currently an experimental feature that needs to be enabled from the Ghost admin.

Publishing Ghost sites using Netlify

Netlify is a service that automates deployment of Jamstack sites. It provides a continous delivery workflow for publishing Jamstack sites. The Ghost has an integration plugin for Netlify that allows Ghost sites to be deployed using the Netlify service.

To use Netlify, the user has to publish the markdown files containing the website content to an online version control repository such as GitHub or BitBucket. When the user makes a commit, Netlify fetches the markdown files and runs a command which generates the static html pages. The html pages are then deployed to a content delivery network (CDN). Netlify allows custom domains, HTTPs and has a free plan.

Conclusion

Ghost is light weight CMS built on modern web technologies. It is a headless CMS and can be integrated with other frameworks and services. It is good option for use as a simple blog, single page applications or small company website.

Building Ionic Cordova applications for Android on Debian 10

Wed, 27 Nov 2019 00:00:00 GMT

Introduction

Cordova is a platform that allows development of Mobile applications using HTML, CSS and JavaScript. These applications can be run on all major mobile platforms, which are Android, IOS and Windows Phone.

The Cordova platform essentially creates a native application for the configured platform. The native application uses a WebView to run the application. A WebView is a browser run-time component for native mobile applications. A WebView runs HTML, CSS and JavaScript code just like a desktop browser.

Applications created using Cordova are termed as hybrid mobile applications, because they are a combination or hybrid of native applications and browser based applications. Native applications are applications that are installed on a mobile device. Browser based applications are applications that are accessed using a mobile device’s web browser

The Android emulator is an application that allows running different types of Android based devices in a simulated environment. The emulator works just like the actual device. It allows debugging and testing Android applications.

The Cordova platform allows building and deploying applications written in HTML, CSS and JavaScript to real mobile devices or emulators. Each platform such as Android or IOS has its own emulator. Cordvoa also provides a live reloading feature. This feature allows automatically updating an application in a real device or emulator once it has been updated by the user. This saves the developer a lot of time and effort.

Cordova applications can be easily tested on emulators. The emulator needs to be installed and configured for the required platform. The emulator for IOS platform is only available on IOS devices.

For the Android platform installing the emulator requires installing the Android Studio. On Linux based systems, the Android Studio requires a graphical desktop environment such as Gnome or KDE. The Android studio has many features and may not be needed during development of the Cordova application. In this article I will describe how to run an Ionic Cordova application on an Android emulator without installing Android Studio.

Installing the Cordova Android requirements

In order to build and deploy Cordova applications on an Android emulator, we need the to install Android SDK, Gradle and Java Development Kit (JDK) version 8. Cordova only supports JDK 8. To find out if the requirements have been correctly installed, enter the command cordova requirements from the root folder of the Cordova application.

The Apache Cordova Platform Guide describes which version of Cordova supports which Android versions. It also describes how to install and configure the requirements for the Android platform.

Installing JDK 8

To install JDK 8, simply download the JDK 8 zip file from the Java Oracle website. Download the file and extract it to /opt, or some other folder of your choice.

Next set the JAVA_HOME environment variable in .bashrc located in your home folder. Also add the directory containing the java binary file to the PATH environment variable. This can be done by adding the following commands to the .bashrc file:

export JAVA_HOME=/opt/jdk1.8.0_231/
export PATH=/opt/jdk1.8.0_231/bin:$PATH

Installing Gradle

Next install the latest version of Gradle by using sdkman or by manually downloading the binary package. To install the binary package, simple download and extract the gradle file to the folder /opt.

Next add the directory containing the gradle binary to the PATH environment variable. This can be done by adding the following command to .bashrc file: export PATH=/opt/gradle-6.0.1/bin:$PATH

Installing Android Development Tools

Next download and extract the Android Development Tools package to /opt folder. Next set the ANDROID_HOME and ANDROID_SDK_ROOT environment variables.

Also add the folder containing the sdkmanager and emulator binaries to the $PATH environment variable. This can be done by adding the following to .bashrc file:

export ANDROID_HOME=/opt/sdk-tools-linux
export ANDROID_SDK_ROOT=$ANDROID_HOME
export PATH=$ANDROID_SDK_ROOT/tools/bin:$ANDROID_SDK_ROOT/platform-tools:$ANDROID_SDK_ROOT/emulator:$PATH

Installing the Android SDK and Platform Tools

Next install Platform Tools and the Android Platform SDK. Install the version of the Android Platform SDK that is supported by the installed version of Cordova. Also install the Android SDK Build Tools, Android Emulator, Android SDK Tools and a System Image for your Android Virtual Device (AVD). The AVD will be used by the Android emulator. These packages can be installed with the sdkmanager tool, using the following commands:

sdkmanager "build-tools;28.0.0" "platform-tools" "tools"
sdkmanager "system-images;android-28;default;x86_64"

Creating AVD and running the Android emulator

After that create an Android Virtual Device (AVD) using the command:

avdmanager create avd -n emulator-name -k "system-images;android-28;default;x86_64"

Once the AVD has been created, start the emulator using the newly created AVD. This can be done using the command: emulator -avd emulator-name

Next run the command: cordova requirements to check if the requirements have been correctly installed. After that add the android platform to your Cordova project using the command: ionic cordova platform add android. Next generate the Android app using the command ionic cordova build android. After that deploy the app to the Android emulator using the command: ionic cordova emulate android. You should now see your app running on the emulator inside a WebView.

Conclusion

The Ionic Framework can use either Apache Cordova or Capacitor for deploying applications written in HTML, CSS and JavaScript to mobile platforms.

In this article I have described how to prepare the development environment so it can be used to deploy Ionic applications to Android platform using Apache Cordova.

Securing websites using Content-Security-Policy and X-Frame-Options

Wed, 02 Oct 2019 00:00:00 GMT

Introduction

Websites are playing an increasingly important role in society. They are no longer used casually. Over the last few years websites have evolved from an amusing hobby to a platform for delivering important content. The content delivered by websites is being used to make important decisions. It is therefore very important to ensure that website content is securely delivered.

Ensuring the security of website content is no easy task. To secure a website we need to secure the back-end as well as the front-end of the website. There are many options for securing websites. The website security options that I will describe in this article are X-Frame-Options and Content-Security-Policy. X-Frame-Options and Content-Security-Policy are both HTTP response headers which are sent by the server to the web browser. They are mostly used to specify which domains are allowed to show content on the website.

X-Frame-Options

X-Frame-Options is a HTTP response header. The web server which sends this header indicates to the browser the domains that are allowed to embed content served by the web server. It can be used to prevent unauthorized websites from embedding content in iframes. The “allow-from” value of this header indicates the specific domain which is allowed to embed content in iframe.

Unfortunately we can only specify one domain name with the “allow-from”. A work around to this limitation is mentioned in the RFC for X-Frame-Options. The workaround is to include the domain name in the iframe source url. The domain name is the name of the domain that has the iframe. The server that delivers the iframe content needs to check this domain value. If the domain is allowed to embed content, then the server can send the “allow-from” http response header containing the domain name.

Before using this header, Browser compatibility of X-Frame-Options, should be checked. The “allow-from” is not supported by Google Chrome.

Content-Security-Policy (CSP)

The X-Frame-Options HTTP response header is an old method of securing content. A newer method is the HTTP Content-Security-Policy (CSP). It consists of HTTP headers that allow website administrators to control the resources that a browser is allowed to load for a give page. It prevents websites from displaying content from unauthorized domains. It helps to protect websites against Cross Site Scripting (XSS) attacks.

The frame-ancestors directive allows us to specify the domains that are allowed to embed content in iframes. It works similarly to X-Frame-Options allow-from, but it allows multiple domain names with wildcards to be specified. Unfortunately the frame-ancestors directive is not supported by Internet Explorer.

The Content-Security-Policy header supports several other directives. Some of them are:

connect-src. Specifies the domains that are allowed to be used as script sources
font-src. Specifies the domains that are allowed to serve font files
frame-src. Specifies the domains that can be used as source for iframes
img-src. Specifies the domains from which images may be loaded
media-src. Specifies valid sources for loading media using the audio, video and track elements
script-src. Specifies valid sources for JavaScript. It includes inline JavaScript and event handlers
style-src. Specifies valid sources for CSS Stylesheets

If a server sends both X-Frame-Options response header and the Content-Security-Policy response header, then the information in the Content-Security-Policy response header is given preference over X-Frame-Options.

Conclusion

Both X-Frame-Options and Content-Security-Policy response headers should be implemented. Using both headers will allow the content security policy to be enforced by all popular web browsers. Websites that implement a good content security policy are likely to have higher rankings in search results.

Recovering innodb data after accidental removal of log files

Wed, 02 Oct 2019 00:00:00 GMT

Introduction

The role of Full Stack web developers is becoming increasingly popular. A Full Stack Web Developer is one who has a working knowledge of all components involved in website development. This includes front-end programming, back-end programming and server administration.

Server administration is a task that requires a lot of patience. Very often stressed out developers make the mistake of deleting important files. In this article I will describe my experience of restoring innodb database tables after accidentally removing innodb log files.

Innodb database table recovery

A server that I was in-charge of started crashing after every few hours. Initially the problem appeared to be caused by SQL injection attacks on a website. I was able to stop these attacks by configuring Fail2Ban. Even after configuring Fail2Ban, the server did not stop crashing.

I checked the running processes. The list showed two instances of MySQL server. Also the MySQL error log file showed that several database tables had crashed. Unfortunately I panicked and made the wrong decision of deleting the log files used by MySQL database server.

The problem

I had deleted the innodb1, ib_logfile0 and ib_logfile1 files. These files should not be removed as they are used by the innodb engine to store important table meta data. After removing the files all innodb database tables were inaccessible.

Fortunately the innodb_file_per_table option was enabled in MySQL server configuration. This option was enabled by default. When enabled, each innodb table uses one tablespace file with .ibd extension and a table structure file with .frm extension. innodb also uses a system table space, which is saved in the innodb1 file. I had removed the system tablespace, but fortunately, the tablespace files for each table were present.

The solution

Recovering crashed innodb tables is a common problem. The article Recover MySQL InnoDB tables without ibdata1 file, is an excellant guide on how to recover crashed innodb tables.

It was used as the main reference for this article. Other useful related articles are: MySQL InnoDB lost tables but files exist, InnoDB: Error: tablespace id in file, Troubleshooting InnoDB Data Dictionary Operations and Moving or Copying InnoDB Tables.

Recovering the table schema

To recover the data, first the MySQL data directory which is /var/lib/mysql had to be backed up. Next the table schema for each innodb table had to be recovered using the mysqlfrm tool. This tool is provided by the mysql-utilities package.

The tool parses the .frm file for the innodb table and displays the CREATE TABLE statement to console. The CREATE TABLE statement can be copied to a new database. In this way the structure for all the innodb tables can be copied to a new database. Once the schema for the table is recovered, the data for the table needs to be recovered.

Recovering the table data

The data for the table is saved in a file with .ibd extension. To recover the table data, first the tablespace of the table needs to be discarded using the SQL query: ALTER TABLE table-name DISCARD TABLESPACE. Next the .ibd files for the table needs to be copied from the backup folder to the folder of the database that will contain the recovered data. Once this is done, the data for the table needs to be imported using the SQL query: ALTER TABLE table-name IMPORT TABLESPACE. Next issue the SQL query: SELECT * FROM table-name should show the table data.

The table can now be exported using mysqldump tool or PhpMyAdmin. The original database table, which is corrupted can be dropped and the new table can be imported. This needs to be done for each innodb table.

Automating the data recovery

To make the task for recovering the data easier, I wrote a PHP script which recovers all innodb tables in a given database.

To use the script do the following:

Download the script using the command: wget https://gist.github.com/nadirlc/7bcf0fcc2b85e6fc15812b46525f51da

Rename the script to restore_innodb.php. To see the script options run: PHP restore_innodb.php

Run the script with the correct options. For example:

php restore_innodb.php --src-db="mysql"
--backup-folder="/root/scripts/data-backup/"
--mysql_data_dir="/var/lib/mysql"
--log_file="/root/scripts/restore_innodb.log"
--user="root" --password="root-password"

just replace the line: $src_db = “mysql”;, with the name of the database that contains the crashed tables. Also replace the line:

$backup_folder = "/root/scripts/data-backup/";

with the name of the folder that contains the backup data. The backup data should include the .frm and .idb files.

The script will replace the corrupted table with the restored table. The script may not work for all cases, because the .ibd files may be corrupt.

Conclusion

Innodb engine has some useful features. For example Innodb database engine supports transactions and row level locking. It is also ACID (Atomicity, Consistency, Integrity and Durability) compliant. It provides supports for foreign keys and also has faster write access than the MyISAM database engine.

However Innodb tables can easily become corrupted because of improper database server shutdown. In most cases the Innodb engine can repair corrupt tables automatically using the Innodb log files, but in some cases the Database Administrator needs to recover the data manually.

Configuring the GamiPress WordPress plugin

Thu, 19 Sep 2019 00:00:00 GMT

Introduction

GamiPress is a WordPress plugin that allows adding Gamification features to a WordPress website. It is free and open source. The main features of GamiPress are free but some of the addons, service plans and assets are priced.

GamiPress provides a system for awarding points based on actions taken on the WordPress website. GamiPress supports Points, Achievements and Ranks for awarding users. It also allows displaying reports on the points, ranks and achievements earned by the users.

GamiPress provides integrations with a number of well known WordPress plugins such as Forminator, LearnDash, WP-PostRatings, BuddyPress and more. Most of these integrations are free. GamiPress also provides add-ons, such as frontend reports, Conditional Emails, Extended REST API, Coupons and more.

The GamiPress website and the GamiPress documentation allow easily getting started with GamiPress.

Installation and Configuration

GamiPress is installed normally like other WordPress plugins. It may be installed from the WordPress plugin manager or by uploading the plugin. Once installed it needs to be configured so it is integrated with the website. Configuring the plugin involves creating Point Types, Achievements and Ranks. To add more features, add-ons and assets may be configured.

Point Types

GamiPress allows creating Point types. Point types are used to automatically award users based on their interaction with the website. Point types can have any name. For example coins, gems, credits, echoes, pulses, impact etc. When creating a point type, we need to specify the point type name, plural name and slug. We also need to specify how the points should be automatically awarded and deducted.

Points for a point type are awarded and deducted in a similar way. We can specify how many points should be earned when a certain action has been taken by the user. We can also specify how many times the action should be taken by the user and also the time frame within which the action can be taken. We can specify the maximum number of times that points can be earned for this action. We can also specify a custom label for the action.

For example, we can specify that when a user has commented on a post 4 times during a week, he should be awarded 10 points. Similarly for deducting points, we can specify that when a user deletes a post, 1 point should be deducted.

Achievement Types

GamiPress allows creating Achievement Types and Achievements. Once an Achievement Type has been created, a sub menu item with the name of the Achievement Type is added under the Achievement menu. Clicking on this sub menu allows us to add the Achievement Types. For example we can create an Achievement type called Skills. We can then add skills such as Leadership Skills, Communication Skills, Computer Skills etc.

When creating an instance of an Achievement type, for example when adding a new skill, we can specify the number of points to be awarded for achieving the skill. We can also specify how the skill can be earned. A skill may be earned by completing steps, by acquiring certain number of points of a certain point type, by reaching a rank or by awarding manually. we can also specify how many times the skill can be achieved. We can also specify that the skill can be earned by spending a certain number of points.

If we choose to allow the skill to be earned by completing steps, then we can add the steps that need to be taken in order to achieve the skill. For example logging in to the website 5 times, creating a post 4 times, commenting on a post 2 times, getting comments on a post 10 times etc. We can also specify if the steps must be completed in sequence.

When adding a new achievement for an achievement type, we can also specify the template that should be used for displaying the achievement. In this template we can choose whether to show list of users that have earned this achievement.

Rank Types

GamiPress allows creating Rank Types and Ranks. Rank Types and Ranks are created similarly to Achievement Types and Achievements. We first need to create a Rank Type. For this Rank Type we can create ranks.

When creating a rank, we need to specify the rank name, plural name and slug. We also need to specify how the rank should be earned. Ranks with the lowest priority do not have any requirements. Ranks with higher priorities need to be earned by completing steps. This is similar to the steps required for earning an achievement. We can also specify that the rank can be earned by spending certain number of points of a certain point type.

Events

An event is an action that needs to occur in order to complete an achievement step, rank requirement or award/deduct points. The main event types are: WordPress Events, Site interactions, Integrations and GamiPress events.

WordPress Events

WordPress events are events such as creating a new post, posting a comment, logging in to the website, deleting a post etc. These events are built in to GamiPress and are supported in the default GamiPress installation.

Site Interaction Events

Site interaction events include visiting the website, visiting a post, getting visits to a post etc.

Integrations events

Integrations events are events supported in an integration of GamiPress. GamiPress integrations are basically WordPress plugins that allow Gamifying external plugins. For example the GamiPress Forminator Integration allows awarding points to users based on their interactions with forms and quizzes, that have been created using the Forminator WordPress plugin.

This integration adds several events to GamiPress such as events related to forms, quizzes and polls. For example we can specify the number of points that should be awarded to a user when a certain form has been submitted with a certain field value by the user.

GamiPress events include events such as unlocking a specific achievement type, earning a certain number of points, reaching a rank etc.

Blocks, Shortcodes and Widgets

GamiPress blocks are provided in the Gutenberg WordPress editor. For example the User Points block allows displaying the current or specific user balance. The User Earnings block allows displaying list of user earnings. These blocks may be displayed anywhere where blocks are supported. Such as posts, pages, custom posts etc.

GamiPress also provides short-codes for displaying information such as a single achievement, user ranks, user earnings etc. For example the short-code:

[gamipress_achievement]

renders a single achievement. Each short-code supports several attributes that allow customizing the short-code output. For example:

[gamipress_achievement id="1" title="yes" link="yes" thumbnail="yes" excerpt="yes" times_earned="yes" steps="yes" toggle="yes" earners="no" layout="left"]

A short-code button appears on the classic WordPress content editor. It allows customizing the short-code. The Gutenberg editor provides blocks for each short-code.

GamiPress provides several widgets that allow displaying information such as the user earnings, user points, user achievements etc. Each widget supports several customization options.

Conclusion

GamiPress is a comprehensive plugin that allows Gamifying WordPress websites. It has many features including integrations, add-ons and service plans. It is easy to configure and customize.

It may be used to Gamify websites such as online education courses, job recruitment websites, e-commerce websites, support forums etc. It helps to improve user engagement with websites.

Preventing SSH and SQL injection attacks using Fail2ban

Tue, 10 Sep 2019 00:00:00 GMT

Introduction

Fail2Ban is an intrusion detection and prevention tool for Linux operating system. It is written in Python language. It works by checking log files using regular expressions. Entries that match the regular expression are banned by adding the IP address to the system firewall.

Fail2Ban is usually used to prevent Brute force SSH attacks. It can also be used to prevent attacks on other services such as web server, ftp server, mail server or even a custom service.

Installation

Fail2Ban can be installed on Debian 9 (stretch) using the command:

sudo apt-get install fail2ban

This configures fail2ban to automatically start on system boot. Fail2ban service can be controlled using the fail2ban-client command. For example the command:

fail2ban-client start

starts the Fail2Ban service. The command:

fail2ban-client status

lists the active Fail2Ban jails.

Configuration

Fail2Ban uses the concept of filters, actions and jails. A filter defines the regular expression used to search for invalid entries in a log file. An action is taken to ban an IP address that matches a filter. The default action is to add the IP address to the system firewall.

A jail defines the log file to be checked as well as other settings such as the maxretry, bantime and findtime. maxretry defines the maximum number of times the server can be accessed from an IP address, before it is banned. findtime defines the time window within which login attempts are made. For example if 5 login attempts are made within a 10 min findtime value, then the IP address is banned.

By default Fail2Ban is configured to ban failed SSH login attempts. Fail2Ban is pre-configured with jails for various services. However these jails need to be activated and customized if required.

For example to activate and customize the nginx-botsearch jail, we need to create the file: /etc/fail2ban/jail.local. Alternately we can create this file in /etc/fail2ban/filter.d folder. This file should have following contents:

[nginx-botsearch]

enabled  = true
logpath  = /var/log/nginx/access.log
filter   = filter-name
maxretry = 2
port     = http,https

The enabled line indicates that the filter should be active. logpath is the path to the Nginx access log file. filter indicates that the filter file, filter-name.conf should be used. The filter file should be defined in /etc/fail2ban/filter.d folder. The following is a sample filter file:

# Fail2Ban filter for WordPress
#
#

[Definition]

failregex = ^<HOST> - - \[.*\] "GET \/company\.php.*select count.* HTTP.*"$

The failregex line indicates the regular expression to use to check for invalid entries in the Nginx access log file. Entries that match the regular expression are banned.

After the filter and jail has been configured, we need to restart the Fail2Ban server using the command: fail2ban-client stop followed by fail2ban-client start. If the regular expression is working, then we should see some entries in the Fail2Ban log file which is: /var/log/fail2ban.log.

Since it can be difficult to develop and test a regular expression for a certain log entry, we can use the fail2ban-regex command for testing our regular expression.

The command takes two arguments. The first one is the log file entry. The second argument is the regular expression. The following example shows how to test that a regular expression correctly matches a given log entry. Note that the code should be entered on one line.

fail2ban-regex '162.144.34.147 - - [09/Sep/2019:10:02:55 +0000]
 "GET /company.php?id=1019999999999 or
 (select count(*)from(select 1 union select 2 union select 3)x
group by concat(mid((select vErsion()
  limit 0,1),1,0),floor(rand(0)*2))) -- x=x HTTP/1.1" 301 169 "-" "-" "-"'
'^<HOST> - - \[.*\] "GET \/company\.php.*select count.* HTTP.*"$'

Conclusion

Fail2Ban is a very useful tool for preventing server intrusions over a network. It works by checking log files and banning IP addresses that produce invalid entries in the log file. It can also be used to secure network based custom services and applications.

The guide Use Fail2ban to Secure Your Server was used as reference for writing this article. It is an excellent resource on how to install and configure Fail2Ban.

Running Graphical applications within LXD containers

Wed, 21 Aug 2019 00:00:00 GMT

Introduction

LXD containers provide an isolated environment, which is separate from the main host. Sometimes we may need to run GUI apps from the container. For example during development of desktop applications.

In the article I will describe how to run GUI apps from within LXD containers. The code for the app is installed on the container, but the app itself runs on the host using the display server of the host. The article How to run graphics-accelerated GUI apps in LXD containers on your Ubuntu desktop was used for reference.

Configuration

Configuring a LXD container so it runs GUI apps requires sharing the host display device with the container and adding a GPU device to the container. Both steps require a mapping of user ids between the host and the container.

The id of the user that will run the GUI apps on the container needs to be matched with the id of the user that is logged in to the host. To test the configuration we need to install a sample GUI app on the container. The sample app is called xclock and it displays a small analog clock in a window. The following configuration steps are involved:

Mapping user ids

This involves creating a mapping between a user on the container and a user on the host. First we need to define a mapping between the root user on the container and the user on the host. This can be done by running the following command on the host:

echo "root:$UID:1" | sudo tee -a /etc/subuid /etc/subgid

The command adds a mapping entry to the /etc/subuid and /etc/subgid files.

Next we need to define a user id mapping for our container. Assuming our container is called Projects, the following command creates a mapping between user id $UID on the host and user id 1001 on the container:

sudo lxc config set Projects raw.idmap "both $UID 1001"

The command allows devices and files belonging to the host user to be used within the container. For the changes to take affect, we need to restart our container with the command:

sudo lxc restart Projects

User id 1001 should be replaced with the user id of the user that will run the GUI apps on the container. $UID is an environment variable that contains the user id of the user logged in to the host.

Next we need to share the X11 display socket and X11 authentication file with the container. We can do this with the following commands:

sudo lxc config device add Projects X0 disk path=/tmp/.X11-unix/X0 source=/tmp/.X11-unix/X0
sudo lxc config device add Projects Xauthority disk path=/home/nadir/.Xauthority source=${XAUTHORITY}

After running the above commands, the container should be restarted. We should then be able to see the folder: /tmp/.X11-unix/X0 and the file: /home/nadir/.Xauthority on the container. Both the folder and files should be owned by user id 1001 on the container.

Add GPU device to container

The next step is to add a GPU device to the container. This can be done with the following commands:

sudo lxc config device add Projects gpu gpu
sudo lxc config device set Projects gpu uid 1001
sudo lxc config device set Projects gpu gid 1001

Test the configuration

Next we need to run a sample GUI app on the container. To do this, we need to install the x11-apps package using the command:

sudo apt-get install x11-apps

The x11-apps package contains the xclock sample GUI app.

Before running the sample app, we need to set the DISPLAY environment variable on the container. It should be set to the same value as the DISPLAY variable on the host. To get the value of the DISPLAY variable on the host, run the command:

echo $DISPLAY

on the host. The value of this variable is usually “:0”.

Next run the command:

export DISPLAY=:0

on the container. It sets the DISPLAY variable to “:0”. Place the command in ~/.bashrc file so next time the user logs in the DISPLAY variable is automatically set.

Run the sample GUI app using the command:

xclock

It should open a small window containing an analog clock. The window will be opened on the host.

Running Android emulator within LXD container

To run an Android emulator within a LXD container, we need to first install the Android SDK on the container. See the article: Building Ionic Cordova applications for Android on Debian 10 without Android Studio on how to install Android emulator.

Once the Android SDK has been installed, enable KVM hardware virtualization on the LXD container. KVM hardware virtualization is required for running Android emulators. To enable the KVM support enter the following commands from the main host that is running the LXD container:

sudo lxc config device add ContainerName kvm unix-char path=/dev/kvm
sudo lxc config device set ContainerName kvm uid UID
sudo lxc config device set ContainerName kvm gid GID

The first command will add the KVM device to your LXD container. The last two commands will set the user and group ids of the KVM device on the LXD container. Change the UID and GID to the user and group id of the user that will run the emulator on your container. After that shut down and start your LXD container. You should now be able to run Android emulator from your LXD container.

Conclusion

Developers often need working environments that are separated from their main computer. LXD containers provide a lightweight environment for development work. However LXD containers do not support running GUI apps by default. Fortunately there is a simple workaround that I have described in this article, which allows GUI apps to be run from LXD containers.

5 Best JavaScript Editors: Developers Rank the Best Tools Worldwide

Sun, 18 Aug 2019 00:00:00 GMT

Introduction

JavaScript has continued to grow in importance over the last decade. In fact, according to StackOverflow’s Insights, JavaScript has been the most popular programming language for the past 7 years, edging out popular languages including Python, C# and Java. The State of JavaScript survey, which includes responses from over 20,000 developers, reveals that five JavaScript editors stand out accounting 95% of all usage!

Editors

1. Visual Studio Code

VSCode is the dominant leader in online JavaScript editors. Featuring cross platform support on Linux, macOS and Windows, VSCode has built in code completion for your node.js modules and JavaScript code. As expected with Microsoft, Typescript is a first class citizen. Git is seamlessly integrated, meaning you can make commits, reviews diffs and more in real-time all without leaving the editor. Visual Studio Code is a lightweight code editor that was built from the ground up for speed. Auto-completion goes beyond function completion and also offers documentation and function argument information as you develop. Microsoft has done an excellent job with community engagement and VSCode has a vibrant developer community creating powerful extensions which save additional time. Sought after features such as live preview and chrome debugging with devtools are easily accessible through the Visual Studio Marketplace. In addition, if you really want to get under the covers, the entire text editor source code is hosted on GitHub as open source software.

2. Sublime Text

A commercial editor with a large user base, developers find the $80 fee for Sublime Text well worthwhile due to its speed (in part derived from it being written in C++). As with VSCode, there is a great developer and plugin community that has filled in the gaps creating a powerful editor environment for JavaScript. Often referred to as a midpoint between bloated IDE and lean editors such as VIM, Sublime opens files fast and leans on plugins via package control to make it a great experience for JS development. Leveraging Babel for intelligent syntax highlighting /smart code completion and gitgutter for diffs and pushes, Sublimelinter for seamless ESlint and JShint tie-ins makes Sublime a fast, capable and lightweight JavaScript editor.

3. VIM

One of the earliest and still very popular text editors, VIM is highly customizable and configurable with exceptional keyboard shortcut support. It’s long history and keyboard dominated interface have produced a developer community with must have plugins like prettier, ALE and powerful command line support enabling you to leverage ESLint and Flow. VIM is incredibly customizable and no two JavaScript developers use it in the same way. If you started early with VIM, you may be very productive in it, but for those new to JavaScript development, VSCode, Sublime Text or Jetbrains WebStorm is a better choice.

4. WebStorm

Jetbrains popular WebStorm is a paid editor that has a loyal following and is regularly updated. Branded, “the smartest JavaScript IDE,” WebStorm lives up to the marketing by offering coding assistance across Node.js, HTML and CSS. It also supports built in assistance with popular JavaScript frameworks including Meteor, Angular, React, jQuery, vue.js and electron. GitHub support is integrated within the IDE and other version control options are supported through official plugins including git integration, Mercurial, Perforce, and Subversion. A built in debugger for Node.js with test integration, tracing and profiling along with seamless command line tool integration round out this IDE. Attention to detail and workflow optimization including automatic saving of file changes is a hallmark of WebStorm. Regular updates with new features and a Early Access Program make the paid subscription worthwhile for many developers.

5. Atom

Marketed as a hack-able text editor for the 21st century, Atom has been growing in popularity. Atom has already established a loyal following and vibrant developer community. Packages give advanced integration options offering real-time collaboration and pair coding, git and GitHub integration and more. A cross platform editor with OS X, Windows and Linux support, Atom runs on Electron and enables deep customization and styling. As a JavaScript developer you can customize Atom using JavaScript, HTML, CSS and Node.js.

Conclusion

Developers are a unique breed, creating thousands of diverse solutions to meet every specific need or niche. Modern editors and IDEs have embraced plugin architectures which allow quick customization and leverage community contributions making it possible for generalized editors such as VSCode to meet more developer’s unique needs. This enables seasoned developers to focus on their JavaScript projects rather than hacking their dev environment while allowing new developers to hit the ground running.

This article was originally published on JSCharting.

Online Collaboration with Slack

Sat, 17 Aug 2019 00:00:00 GMT

Introduction

Slack is a popular online collaboration platform that is widely used by professionals from different fields. It allows instant messaging, voice and video conferencing, file sharing, integration with applications and more.

Slack provides customized solutions for Customer Support, Engineering, IT, Human Resources, Marketing, Project Management, Sales and Task Management. Slack not only connects people and teams but also allows integration of tools and services.

Slack implements a client server model, where the server is hosted by Slack and provided under different pricing options. Slack provides client apps for Windows, Linux and Mac Os. It also has apps for Android and Iphone.

Slack provides a Free Plan, Standard Plan, Plus Plan and Enterprise Grid plan. Slack provides accessing using workspace. A workspace is basically a Slack account with a unique name. Each workspace has its own unique URL. For example if we create a workspace called chat, we get the URL chat.slack.com. From this URL, we can change the workspace settings and also access the web based Slack client.

An open source alternative to Slack is Rocket Chat, which allows hosting the server for free.

Integrations

Slack provides integrations with numerous tools and services. The integrations basically allow external tools and services to automatically post messages to Slack channels. They also allow messages posted using Slack clients to be sent to external tools and services.

BuddySlack

Slack can be integrated with WordPress in different ways. For example, the BuddySlack WordPress plugin allows automatically posting user activities to Slack channels. The BuddySlack plugin is easy to configure. We need to first create an incoming WebHook for our Slack WorkSpace. The incoming WebHook has a channel to which the user activities are posted. It also has a URL, which we need to copy and paste to the plugin configuration. After that all user activities related to BuddyPress should be automatically posted to the Slack channel.

Hybrid.Chat

Hybrid.Chat is a separate service that provides ChatBots and API integrations. The service allows creating ChatBots using a simple three step wizard. The wizard provides HTML/JavaScript code which creates a Chat widget.

The Hybrid.Chat WordPress plugin provides a Chat widget. The widget allows website users to chat. The Chat widget may be integrated with Slack. This allows the website visitors to chat with users on a Slack channel. The messages entered by the website user appear in a Slack channel. Replies may be posted using a Slack client.

Conclusion

Slack is a useful online collaboration platform that is easy to use and integrate. It connects people and also enables integration of tools and services.

Building online communities with Discourse

Fri, 19 Apr 2019 00:00:00 GMT

Introduction

Discourse is a popular discussion application that allows users to discuss topics. It is one of the most popular discussion applications with lots of features and a user friendly interface. To try out the features provided by Discourse, see the Discourse Demo.

Discourse has a standard, business and enterprise edition. These editions are non free. There is also an open source edition that is free to use. Discourse is a useful platform for building communities. In this article, I will describe the features provided by Discourse and also how to install Discourse on a Linux server.

Features

Discourse has the following features:

Main Features

The main feature of Discourse is a forum system. It allows users to post topics. Users can discuss the topics by posting replies. The replies can be moderated by special users. Dynamic notifications allow users to receive email alerts when someone has replied to their post or has mentioned them in a reply.

Discourse is a simple forum, in which replies flow down a single page. No need to click through pages to view replies. The context of a post can be viewed by clicking at the top and bottom of the post. The context reveals the full post conversation. Links in a post automatically expand to reveal additional information without requiring the user to click the link.

Discourse also provides open source mobile applications for IOS and Android.

API, Plugins and Theme components

Discourse can be extended with plugins that enhance the basic functionality of Discourse. There are standard plugins that are provided with all hosting plans. These include a chat plugin, plugins for preventing spam, advertising, displaying mathematical notation in posts and more. The Business and Enterprise plugins include a data explorer which allows running SQL queries against databases, a Sitemap plugin, a topic voting plugin and more.

To install a Discourse plugin, we need to edit the containers/app.yml configuration file and enter the text: “git clone [plugin-repository-url]”. Next we need to rebuild the container using the command: ./launcher rebuild app. The plugin should now show on the plugins list page. The article, Install Plugins in Discourse on the Discourse forums describes how to install plugins.

The Discourse API allows access to Discourse data. It also allows posting topics and replies, managing users and more.

Theme components enhance the functionality of themes. For example the Social Share Component, allows adding custom social share links to the post share form.

The Brand header theme component, allows adding a header to the Discourse with custom links. It can be used to brand the Discourse so it looks like the company website.

To install a theme component, we need to go to the theme component admin page and enter the Git Repository URL of the component. We can also choose from a list of popular theme components. Next we need to add the theme component to the current theme. This last step is very important and sometimes missed by users not familiar with Discourse.

Integrations

Discourse provides useful integration options. It supports Social Login with Facebook, LinkedIn, Twitter, Yahoo, GitHub and more. It also provides Custom Oauth2 and SAML. These allow Discourse to be used as a single sign on provider for websites.

Discourse also supports integrations with Chat providers including Rocket Chat, Slack, MatterMost, Gitter and more. The Chat Integrations allow Discourse to notify chat providers about new posts and replies. It also allows chat transcripts to be posted in topic replies.
Discourse has some useful Advertising and Monetizing options. The Discourse Advertizing plugin allows displaying ads above, below and between content. It supports, Google Adsense, Google Ads, Amazon and Code Fund
Discourse supports integration with Analytics providers such as Google Analytics, Motomo and Adobe Analytics Cloud. The Analytics integration allows Discourse admins to view user statistics.
Discourse also integrates with Crowd Funding providers such as Patreon and Memberful. This allows Discourse to be used as a contributor only community. For example it allows Discourse website managers to sell memberships.
The Discourse GitHub integration is provided by the Discourse Code Review plugin. It allows 2 way integration between GitHub and Discourse. It allows new topics on Discourse to be created from GitHub Commits. It also allows Discourse replies to be posted back to GitHub. For example commits for reviews, follow ups and more.
The Discourse ZenDesk plugin integrates ZendDesk with Discourse. It allows Discourse topics to be converted to ZenDesk tickets. It also allows syncing comments between ZenDesk and Discourse.
The Google Perspective API plugin, allows automatically monitoring the sentiment of the community. It allows warning users before they post potentially offensive content. It also allows scanning posts for abuse. It uses Google’s Perspective Machine Learning API
The Discourse Translator Plugin allows translating posts to different languages using Google Translate or Microsoft Translator.
The WP Discourse Plugin allows integrating Discourse with WordPress. It allows automatically creating a new Discourse Discussion for new WordPress posts. It allows embedding Discourse Discussions in WordPress. It also allows using Discourse as an authentication provider for WordPress or using WordPress as the authentication provider for Discourse.
Discourse’s Akismet Integration allows stopping Spam. The Algolia Integration allows integrating intelligent search. Discourse’s Web Hooks allow integration with hundreds of services provided by Zapier or IFTT

Installation

Discourse is based on RubyOnRails platform. It uses PostgreSQL database. The recommended method for running the open source version of Discourse is with Docker. The Discourse Installation Guide for Cloud Platforms describes how to install Discourse on Digital Ocean or on personal servers. Since the Discourse is run within a Docker container, the installation guide applies to all platforms that support Docker.

We first need to install Docker on the server. See the Docker installation guide for how to install Docker on different operating systems. Next we need to create the folder: /var/discourse and download the Discourse Docker source code to the folder. This can be done with the commands:

mkdir /var/discourse
cd /var/discourse
git clone https://github.com/discourse/discourse_docker.git

Next we need to run the Discourse setup tool using the command: ./discourse-setup. The setup script will ask for the Discourse hostname, email settings, and Lets Encrypt account email. I chose to skip the Lets Encrypt setup, since my installation of Discourse does not require https. The setup script will generate the containers/app.yml Docker configuration file. It will also start the boostrap. The bootstrap process builds the Discourse Docker image and downloads it to the server. It also starts a container from the downloaded image. It takes about 2-8 minutes to complete.

Once bootstrapping has completed, we should be able to access the Discourse from a browser using the host name that we mentioned during the setup. We will then be asked to register an Admin user account. Next we should be able to login to our Discourse instance using the admin user account.

Running Discourse with other websites

If your server hosts other websites on port 80, then you will have to edit the Discourse Docker configuration, so that the Discourse instance can be accessed from a proxy web server, such as Nginx. When accessed through a proxy server, the Discourse server will be accessed over a socket instead of a port.

This configuration is described in the article Running other websites on the same machine as Discourse. For example to access Discourse through Nginx, we need to first stop the Discourse using the command: ./launcher stop app. Next we need to change the container definition given in containers/app.yml. After that we need to create a Nginx virtual host. The virtual host should forward requests to the Discourse instance over a socket.

Next we can restart the Nginx server with the command: service nginx restart. We can then start the Discourse using the command: launcher rebuild app. We will then be able to access the Discourse from a browser. Other websites running on the same server should also be working.

Conclusion

Discourse is an excellent platform for building online communities. It supports integrations with well known platforms and provides many features out of the box. Its installation is simple and well documented. Discourse is supported by a large open source community and is widely used.

Linux Server Security - Quick Tips

Tue, 16 Apr 2019 00:00:00 GMT

Following are some quick tips related to Linux Server Security:

Whitelisting devices and files for Rootkit Hunter

Some programs like postgresql create virtual memory devices under /dev/. Rootkit Hunter may raise warnings for these devices. To whitelist a device under dev the ALLOWDEVFILE directive can be used. This value of this directive is a single device path. Wildcard (*) may be used inside the device path. Multiple instances of the ALLOWDEVFILE directive are allowed.

Some programs such as Odoo ERP can update your /etc/passwd and /etc/group files. RKhuRootkit Hunter may report a warning for these file changes. To whitelist the file the RTKT_FILE_WHITELIST directive can be used. The value of this directive is the full path to the file to be whitelisted. If certain strings within the file need to be whitelisted, then the string name can be appended to the file path, with a colon before the string. e.g RTKT_FILE_WHITELIST=/etc/passwd:postgres will whitelist the postgres user entry inside /etc/passwd file

NodeJs permission denied error

The error: sh: 1: node: Permission denied, usually occurs because the node binary does not allow execution with root privileges. To correct the error we need to run the commands:

npm config set user 0
npm config set unsafe-perm true
sudo npm install -g sm

Configuring SFTP and SSH access for LXC Containers

The Redir command allows redirecting TCP connections. It takes as argument, the listen address and port and the connect address and port. The listen address and port are the IP address and port at which the redir script will listen on. The connect IP address and port are the IP address and port that the redir command will connect to and forward the incoming TCP connection.

The redir command is very useful for forwarding ports. For example it can be used to forward SSH connections to a container running inside a server. The server needs to run the redir command which will forward the SSH connection to the SSH server running on the container.

The following command may be used: redir —laddr=0.0.0.0 —lport=1001 —caddr=192.168.1.10 —cport=22 &. The ’&’ at the end causes the redir command to run in the background. It listens on port 1001 and all available IP addresses on the server. It redirects the TCP connection to IP address 192.168.1.10 and port 22, which is the standard SSH port. The SFTP/SSH client would need to establish a connection on port 1001 and IP address of the server.

PrivateTmp systemd option prevents access to tmp folder

Systemd startup configuration files use the option PrivateTmp to prevent processes from sharing tmp folder. When this option is set to true, a private tmp folder is created for each process. So if a process attempts to write to the /tmp or /var/tmp folder, the changes will be made only to the process’s private tmp folder and not the system wide /tmp and /var/tmp folders. This option may be set to true in the system startup file for Apache web server on Ubuntu and Linux Mint. For example in: /lib/systemd/system/apache2.service.

Fixing Rootkit Hunter lwp-request warning

RKhunter may give the following error about lwp-request script:

Warning: The command '/usr/bin/lwp-request' has been replaced by a script: /usr/bin/lwp-request: Perl script text executable

This warning may be ignored because lwp-request is a valid Perl script. It can downloaded from the lwp-request page on CPAN. To supress the warning we need to whitelist the lwp-request file. We can do this by adding the following configuration option to /etc/rkhunter.conf. SCRIPTWHITELIST=/usr/bin/lwp-request. This was suggested in an answer to the question: rkhunter gives me a warning for “/usr/bin/lwp-request” - what should I do? [Debian 9] on unix.stackexchange. It was orginally suggested in the article rkhunter warning [SOLVED] on the Linux Mint forums.

Rootkit Hunter SSH version warning

Rootkit Hunter may give the following error related to root user login configuration:

Warning: The SSH and rkhunter configuration options should be the same:
         SSH configuration option 'PermitRootLogin': prohibit-password
         Rkhunter configuration option 'ALLOW_SSH_ROOT_USER': no

This warning is given, when the root user configuration in the SSHD server configuration is different from the root user configuration in /etc/rkhunter.conf or /etc/rkhunter.local.conf. To suppress this error, we need to set the ALLOW_SSH_ROOT_USER in the Rootkit Hunter configuration to the same value that we set for the PermitRootLogin configuration in /etc/ssh/sshd_config.

Preventing WordPress Spam

Tue, 16 Apr 2019 00:00:00 GMT

Introduction

WordPress is a very popular platform for publishing websites. Almost all features that one can expect from a website are provided in the form of open source plugins and themes. Adding new features to a website is as easy as installing a plugin and configuring it. Highly complex websites based on WordPress can be constructed by installing and configuring the correct plugins and themes.

Given the wide range of available plugins, developing complex web sites based on WordPress may seem like an easy task. But the fact is that developing a complex website is never easy. Without proper planning and attention to detail, an application cannot achieve its required purpose. Very often, Web Developers tend to focus on the features of a website and overlook other important aspects of a website such as its performance and security. Website security is a critical issue. One that is often overlooked by the people who plan and develop the website. Recently I contributed to the development of a website based on WordPress. The website had numerous features, but was not well secured. The security problems with the website affected the entire server and caused big problems for the server administrator. I will share my experience in this article.

Purpose for developing websites

A website can serve many purposes. It can be used as an intranet, as a platform for spreading information about a topic. It can be used as a source of entertainment etc. The point is that a website needs to have a purpose. And it should always be achieving the purpose for which it was built. If a website is not achieving its purpose, it should be brought down. It should not be left accessible to everyone on the internet. An indication that a website is not achieving its purpose is that it is not being updated or being accessed. A website that is not being maintained can develop security problems. This is because the environment in which the website exists, which is the server and the internet are constantly changing.

Problems with neglected websites

Websites that are not achieving any real purpose are a problem for the company and for the system administrators. The main reason why such websites exist is usually a lack of cooperation between system administrators, website developers and managers. The website that I worked with was based on WordPress. It had the latest commercial plugins that provided, forum, blog, learning system, job management, resume builder etc. The website was developed without proper consultation between managers, developers and designers. As a result due consideration was not given to the security of the website. The website was abandoned shortly after its development. For months it was not updated or even checked by the system administrators and manger.

Problems caused by open user registration

A major security problem with the website was that it allowed anyone to register. While this feature is present in many websites, it needs to be carefully implemented. Of course we do not want anyone to register on our website. We do not want bots, scripts and spam users to register. Unfortunately this is what happened with the website. It was attacked with spam user registrations. Within a spam of a few weeks, over 28,000 spam users and over 45,000 spam Buddy Press groups had registered on the website.

Server monitoring

We would never even have known about the problem. The first sign we had of a problem was high CPU usage on our server which was hosted on Google Cloud Platform (GCP). At first we thought this was because of too many applications and insufficient CPU. So we tried to fix the problem by increasing the CPU from 2 to 4. We also removed unused applications based on Node.Js and Docker which are known to consume server resources. However this did not fix the problem and the server consistently had 100% CPU usage. This was a big problem because important company websites were hosted on this server.

The htop command showed high CPU usage by the MySQL and Apache processes. Running the show processlist command from the MySQL prompt showed over 20 SQL queries being run on a certain database constantly. The web server access log showed almost non stop access to a certain website. The overused website was using the overused database. A quick look at the database table showed a large number of users and groups.

Removing Spam users

The Bulk Delete WordPress plugin allows deleting multiple users at once. I used this plugin to delete the Spam users. However the plugin did not work as expected and gave an error. The web server error log showed the following error:

2019-04-09T15:17:11.259450Z 123 [Warning] InnoDB: Table mysql/innodb_index_stats has length mismatch in the column name table_name. Please run mysql_upgrade

[Tue Apr 09 11:32:17.015235 2019] [:error] [pid 3182] [client 192.168.1.1:43080] WordPress database error Table 'performance_schema.session_variables' doesn't exist for
query show variables like 'max_allowed_packet' made by do_action('wp_ajax_nopriv_wordfence_doScan'), WP_Hook->do_action, WP_Hook->apply_filters, wordfence::ajax_doScan_c
allback, wfScan::wfScanMain, wfScanEngine->go, wfScanEngine->doScan, wfScanEngine->scan_knownFiles_main, wordfenceHash->run, wordfenceHash->processFile, wfScanner->incre
mentSummaryItem, wfScanner->_maybeSaveSummaryItems, wfScanner->_saveSummaryItems, wfConfig::set_ser, wfDB->getMaxAllowedPacketBytes, wfDB->querySingleRec

Database server problems

I tried to run the mysql_upgrade command as suggested by the article Table ‘performance_schema.session_variables’ doesn’t exist. But the command would give a “Server has gone away” error. It seemed the performance_schema table in the default MySQL database had become corrupted. We were able to fix the problem by exporting all databases hosted by the MySQL server one by one using mysqldump command. Next we completely removed the database server using the command: apt-get purge mariadb-client-10.1 mariadb-common mariadb-server-10.1. This command removed the database binary files, configuration and data. Before removing the database server we took a snapshot of our GCP instance. We then reinstalled the database server and re-imported the databases from backup. Once database at a time. That fixed the problem with the database server. Next we removed all unused databases.

After that we are able to delete the Spam users from database using the WordPress plugin. The user deletion was taking a long time, so the plugin gave timeout errors. We got passed this problem by removing 2,000 users at a time. Unfortunately we could not find a similar plugin for removing the Spam BuddyPress groups. We had to simply empty all the BuddyPress database tables as described on How to COMPLETELY delete, reset or reinstall BuddyPress, please!. This was not a big problem since the website did not have any real users and groups. It had some demo data that could always be imported back if needed. We could not simply redevelop the entire website since a lot of work had gone in to customizing the website theme and layout.

Stopping Spam with ModSecurity and ReCaptcha

We managed to reduce the number of Spam users and groups by installing and configuring ModSecurity. In my article Integrating ModSecurity with Nginx on Debian 9, I describe how to install and configure ModSecurity with our Nginx web server. We configured the ModSecurity in active mode, which means that the suspicious requests were both blocked and logged. The rate at which Spam users and groups were being created decreased a lot. But the users and groups were still being added.

The article How to Defeat WordPress Spam Users: Identify, Delete, and Prevent Future Registrations, gives excellent advice on how to detect and prevent Spam users on WordPress. The article suggested using Captcha to protect the registration and signup forms. We installed the Google Captcha (reCAPTCHA) by BestWebSoft plugin and used it to protect all forms on our website including the registration and signup forms.

Captcha’s are commonly used to protect forms from bots and scripts. They usually require the user to answer a simple question, enter the letters shown in an image or select items shown on an image. Bots and Scripts are usually not able to pass a Captcha. However these Captcha’s are inconvenient for the user and affect the usability of the website.

Google’s new ReCaptcha v3 solves this problem. It does not require any user interaction. It just shows an image to the user which shows at the bottom right. The image scrolls with the users scrolling. It shows a link to the privacy policy of the ReCaptcha. The ReCaptcha works by assigning a score to the user based on the users interaction with the website. This score is calculated using different factors. The maximum score that indicates a human user can be configured in the ReCaptcha settings. The ReCaptcha settings also how statistics on the form submissions that were blocked.

To use the ReCaptcha v3 with WordPress we need to first sign up on the ReCaptcha website. Next we need to copy the access and secret keys. After that we can install a WordPress plugin for the ReCaptcha and configure it with the ReCaptcha keys.

Preventing Spam with a WordPress security plugin

The All in one WP Security and Firewall is a useful security plugin which provides several security related features. It provides a dashboard page that lists the security score of the website. The score is calculated based on configuration of the security options. The plugin allows disabling XML-RPC requests, protecting .htaccess file, wp-config file, blocking and white-listing IP addresses, showing list of users logged in and more.

Even after installing and configuring the plugin, the Spam users were still being added to the website. The only possible explanation is that the plugins installed on the website, which process the registration requests do not apply security checks to the form submission. The only solution was to completely block user registrations using .htaccess file. We used the following code in .htaccess file to achieve this:

&#x3C;IfModule mod_rewrite.c&#x3E;
RewriteEngine On
RewriteCond %{REQUEST_METHOD} ^POST [NC]
RewriteCond %{REQUEST_URI} ^.*/register/$
RewriteRule .* http://127.0.0.1 [L]
&#x3C;/IfModule&#x3E;

The above code allowed the registration page to be accessible, but when the register form was submitted, the user was redirected to the URL: http://127.0.0.1. This completely stopped the Spam user registrations. We also changed the password for all admin users. After that our problems were solved. The Spam users and groups were not being added and the CPU usage was back to normal.

Conclusion

WordPress is an excellent platform for developing websites. However the WordPress developers and the developers of WordPress plugins and themes, need to pay more attention to security. A website may have great features, but if it is not secure, then it is useless.

Developers of websites and managers who supervise the development of the websites should work together to ensure that the website is secure and is achieving the goal for which it was developed.

Integrating ModSecurity with Nginx on Debian 9

Sat, 13 Apr 2019 00:00:00 GMT

Introduction

ModSecurity is a an open source Web Application Firewall (WAF). It can detect as well as prevent attacks to web applications. Initiallly released as an Apache web server module, ModSecurity now supports all major Web Servers including IIS, Nginx and Apache.

It also provides a standalone library called Libmodsecurity that can be used in external applications. Connectors are used to allow external applications to use the Libmodsecurity library. The connectors serve as interfaces between external applications and Libmodsecurity. For example Nginx uses Libmodsecurity through the ModSecurity-nginx connector. In this article I will describe how to install ModSecurity and integrate it with Nginx on Debian Stretch using the ModSecurity-nginx connector.

Installation

ModSecurity can be integrated with Nginx web server using the ModSecurity-nginx connector. The ModSecurity-nginx connector is a Nginx module which uses the Libmodsecurity library. ModSecurity can be enabled for the following contexts: server, http and location. the main Nginx directives provided by the ModSecurity are: modsecurity on | off and modsecurity_rules_file [path to rules file]. Once enabled it checks each request and compares it with the OWASP ModSecurity Core Rule Set (CRS).

The OWASP ModSecurity Core Rule Set (CRS) is a set of attack detection rules which aims to protect Web Applications from a wide range of attacks including the OWASP Top Ten list. The CRS is intended for ModSecurity but may be used by other applications.

ModSecurity may be configured in detection mode or in prevention mode. In detection mode, ModSecurity simply logs the error message to a log file. In prevention mode, the error is not only logged, but the request is blocked and an error response is sent to the sender of the request.

I followed the steps given in the article: ModSecurity and nginx. The article is useful and covers all the steps, but has some short commings. The guide: ModSecurity 3.0 and NGINX: Quick Start Guide, provided by the Nginx website is a comprehensive guide which may be regarded as the official documentation for integrating ModSecurity with Nginx. The main steps involved in integrating ModSecurity with Nginx are on Debian 9 (Stretch) are:

Install Nginx

The installation of Nginx on Debian is documented on the Nginx website. The main steps are:

# Add the repository url to the apt sources list
echo "deb http://nginx.org/packages/debian `lsb_release -cs` nginx" | sudo tee /etc/apt/sources.list.d/nginx.list
# Add the package signing key
curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo apt-key add -
# Verify the package signing key
apt-key fingerprint ABF5BD827BD9BF62
# Update system package list
sudo apt-get update
# Install Nginx
sudo apt-get install nginx
# Verify the Nginx version
nginx -v

Compile the Libmodsecurity

To compile the Libmodsecurity library, we need to first install the dependencies. Next we need to download and compile the Libmodsecurity source code.

Install the dependencies

Use the following command to install the packages required by Libmodsecurity library:

apt-get install libtool autoconf build-essential libpcre3-dev zlib1g-dev libssl-dev libxml2-dev libgeoip-dev liblmdb-dev libyajl-dev libcurl4-openssl-dev libpcre++-dev pkgconf libxslt1-dev libgd-dev

Download the code

Use the following commands to download the correct version of Libmodsecurity source code. Note that the command given in the ModSecurity and nginx article mentions the old source code. Use the command which is mentioned in the ModSecurity 3.0 and NGINX: Quick Start Guide

cd /opt/
# The correct command for downloading the latest Libmodsecurity source code
git clone --depth 100 -b v3/master --single-branch https://github.com/SpiderLabs/ModSecurity
cd ModSecurity
git submodule init
git submodule update

Compile the code

Use the following command to compile the Libmodsecurity source code:

# Generate configure file
sh build.sh
# Pre compilation step. Checks for dependencies
./configure
# Compiles the source code
make
# Installs the Libmodsecurity to **/usr/local/modsecurity/lib/libmodsecurity.so**
make install

Compile the connector

It requires downloading the Nginx source code and the source code for ModSecurity-nginx connector. When downloading the Nginx source code, make sure the version matches the currently installed Nginx version. Even though we just need to compile the ModSecurity-nginx connector, we need the Nginx source code. The following steps are involved:

# Fetch the Nginx source code. See the directory: **http://nginx.org/download/** for all versions of Nginx
wget http://nginx.org/download/nginx-1.14.2.tar.gz
# Extract the downloaded source code
tar -xvzf nginx-1.14.2.tar.gz        
# Fetch the source code for ModSecurity-nginx connector
git clone https://github.com/SpiderLabs/ModSecurity-nginx
# Change directory to the Nginx source code
cd nginx-1.14.2/
# Note the compile time flags used to generate the current Nginx binary
nginx -V
# Compile the Nginx using the compile time flags noted above. Exclude the **add-dynamic-module** options since we only need to compile the ModSecurity-nginx module
./configure [your-compile-time-options noted above] --add-dynamic-module=[path-to-ModSecurity-nginx connector source code]
# Generate the module
make modules
# Copy the module to the Nginx module directory
cp objs/ngx_http_modsecurity_module.so /etc/nginx/modules/

Configure the connector

To configure and enable the ModSecurity-nginx connector, we need to first load the ModSecurity Nginx module. Next we need to download the ModSecurity rules. After that we need to configure the ModSecurity-nginx connector.

Load the module

Add following line to /etc/nginx/nginx.conf:

load_module modules/ngx_http_modsecurity_module.so;

Download the rules

# Create the modsec folder in the Nginx configuration folder. It will contain the ModSec rules
mkdir /etc/nginx/modsec
# Change directory
cd /etc/nginx/modsec
# Download the rules
git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git
# Rename the default ModSecurity rules configuration file
mv /etc/nginx/modsec/owasp-modsecurity-crs/crs-setup.conf.example /etc/nginx/modsec/owasp-modsecurity-crs/crs-setup.conf

Configure the connector

# Copy the default ModSecurity configuration file to the modesec folder
cp /opt/ModSecurity/modsecurity.conf-recommended /etc/nginx/modsec/modsecurity.conf
# Create a configuration file that will be loaded by Nginx. This file will load the ModSec rules configuration file and the ModSec configuration file
vi /etc/nginx/modsec/main.conf
# Add following lines to the file
Include /etc/nginx/modsec/modsecurity.conf
Include /etc/nginx/modsec/owasp-modsecurity-crs/crs-setup.conf
Include /etc/nginx/modsec/owasp-modsecurity-crs/rules/*.conf
# Restart Nginx
service nginx restart
# If you see an error that mentions missing **unicode.mapping** file, then copy the file to the mod sec folder
cp /opt/ModSecurity/unicode.mapping /etc/nginx/modsec

Testing the ModSecurity

To test the ModSecurity Nginx integration, we need to enable the Nginx ModSecurity module for the default virtual host. Add the following 2 lines to /etc/nginx/conf.d/default.conf. The lines should be added under the server_name line:

modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;

Next we need to restart the Nginx server using service nginx restart. We can test the ModSecurity by sending a suspicious request to the Nginx web server. The following command sends a curl request containing a banned user agent:

curl -H "User-Agent: masscan" http://localhost/

The following should be logged to the ModSecurity log file which is /var/log/modsec_audit.log:

ModSecurity: Warning. Matched "Operator `PmFromFile' with parameter `scanners-user-agents.data' against &amp;rarrhkk;variable `REQUEST_HEADERS:User-Agent' (Value: `masscan' )

This indicates that ModSecurity successfully detected the suspicious request. However the request was allowed to pass through to Nginx. This is because ModSecurity operations in detection mode by default, which means it only logs the suspicious requests. To enable ModSecurity to block requests, we need to edit the ModSecurity configuration file: /etc/nginx/modsec/modsecurity.conf and change: SecRuleEngine DetectionOnly to SecRuleEngine On. Next we need to restart Nginx. After that the previous curl command should fail with a 403 error message, indicating that ModSecurity has successfully intercepted and blocked the request. The blocked request will also be logged.

Conclusion

ModSecurity is a very useful tool which allows protecting web applications from malicious attacks. Although it can stop most attacks, it cannot stop attacks that resemble normal user requests. For example spam attacks on WordPress websites may not be stopped by ModSecurity alone. To stop such attacks we can use a tool such as Fail2Ban or a WordPress security plugin like the All In One WP Security & Firewall

For more information about Internet security, read Different types of Internet security threats to think about in 2023.

Graphics - Quick Tips

Fri, 05 Apr 2019 00:00:00 GMT

Following are some quick tips related to Graphics:

Adding transparent background to images using GIMP

To make the background of an image transparent using GIMP:

First open the image using GIMP
Click on Layers -> Transparency -> Add Alpha Channel
Click on Tools -> Selection Tools -> Fuzzy Select
Click on the part of the image that should be transparent. This will highlight a section of the image. Next press delete. This will make the selected portion of the image transparent
After that click on File -> Export As. Next save the image

Drawing rectangles in an image using GIMP

To make the background of an image transparent using GIMP:

Go to Windows -> Dockable Dialogs -> Colors, then select a color
Next go to Tools -> Selection Tools -> Rectangle Select. Next select the rectangular area
Next go to Edit -> Stroke Selection. After that press Enter

Apache and Nginx - Quick Tips

Wed, 03 Apr 2019 00:00:00 GMT

Following are some quick tips related to Apache and Nginx Web Servers:

Apache Multi Processing Modules (MPMs)

The Apache web server supports Multi Processing Modules (MPM) since version 2. An Apache MPM is a module that determines how Apache manages networking connections and output. Apache supports several MPM for different operating systems. The most commonly used MPMs are Prefork, Worker and Event MPMs.

The Prefork MPM is single threaded, which means it only allows one thread per child process. This is not efficient and is not suitable for busy web sites.

The Worker MPM allows multiple threads per child process. It is much more efficient than Prefork MPM. The Event MPM is similar to the Worker MPM. It is more efficient than the Worker MPM since it allows a thread to perform more than one task. It has the lowest resource usage but is considered as experimental for Apache versions before 2.4. For Apache 2.4 and above, the Event MPM seems to be the most suitable MPM.

Calculating the number of simultaneous connections supported by Apache

Modern web browsers make 6 to 8 simultaneous connections to the web server. The number of simultaneous connections supported by Apache web server is determined by the configuration of the MPM that is used.

Generally the maximum number of users that can access a website served by Apache is equal to ServerLimit * ThreadsPerChild. The ServerLimit is the maximum number of processes that are allowed to be created by Apache. The ThreadsPerChild is the number of threads created per child process.

Generate Certificate Signing Request (CSR) for ssl certificates

In order to purchase a SSL certificate, we need to generate a Certificate Signing Request (CSR). The CSR contains information on the domain to be secured and the owner of the domain. To generate a CSR, we can use openssl command. For example the command: openssl req -new -newkey rsa:2048 -nodes -keyout yourdomain.key -out yourdomain.csr, will generate a CSR file and a private key file for the domain yourdomain.com. The CSR file needs to be sent to the certificate authority that will supply the SSL certificate. The private key file should be kept securely, as it will be used by the web server.

Using flags with rewrite rule

The behavior of Rewrite commands can be modified using flags. Some of these flags take arguments. Rewrite supports several flags. The [R] flag causes an external redirection. It means the URL in the browser changes when the [R] flag is used. The [R] flag takes an argument with a default value of 302. This argument is the status code of the HTTP response.

The default behavior of the RewriteRule is to cause an internal redirection to the target path. It means the URL in the browser does not change. The target path is the file system path but it may be given as a URL The [L] flag signals that the processing of the mod_rewrite commands should stop.

Creating and Installing Apache Self Signed Certificate

To generate an Apache self signed certificate, we need to issue the command:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mysitename.key -out mysitename.crt

This will generate a SSL certificate called mysitename.crt and a private key file called mysitename.key. The path to these files can be added to the Apache virtual host configuration.

Redirect http to https with Apache

To redirect http urls to https, we need to add the following directives to .htaccess file:

RewriteEngine On
# This will enable the Rewrite capabilities

RewriteCond %{HTTPS} !=on
# This checks to make sure the connection is not already HTTPS

RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
# This rule will redirect users from their original location, to the same location but using HTTPS.
# i.e.  http://www.example.com/foo/ to https://www.example.com/foo/
# The leading slash is made optional so that this will work either in httpd.conf
# or .htaccess context

Port forwarding with Nginx

To run a website on a different port, port forwarding can be used. For example if a website runs over port 8080, but it needs to be accessed over port 80, then port forwarding can be used. To implement port forwarding in Nginx, the following server block needs to be added to the website configuration file:

server {
    listen 80;
    server_name example.com;

    location / {
        proxy_set_header   X-Forwarded-For $remote_addr;
        proxy_set_header   Host $http_host;
        proxy_pass         "http://127.0.0.1:8080";
    }
}

With the above configuration, Nginx will listen on port 80, but will forward all requests to port 8080 on localhost.

Converting MySQL database to SQlite3 format

Wed, 03 Apr 2019 00:00:00 GMT

SQlite3 is a lightweight relational database that can be used in place of relational database servers such as MySQL and PostgreSQL. SQLite3 databases are commonly used in Mobile applications. In contrast to most relational databases, a SQLite3 database is a single independent file. The file has a .db extension. Most programming languages such as Python and PHP provide libraries for accessing SQLite3 databases.

The mysql2sqlite is a free and open source script that allows converting MySQL databases to SQLite3 format.

To convert a MySQL database to SQLite3, we need to first export the database using PhpMyAdmin or mysqldump. Next we need to run the command:

./mysql2sqlite dump_mysql.sql | sqlite3 mysqlite3.db

This command converts the sql file mysql.sql to mysqlite3.db. The SQLite database can then be accessed from programming languages such as Python and PHP.

Cron Jobs - Quick Tips

Wed, 03 Apr 2019 00:00:00 GMT

Following are some quick tips related to Cron Jobs:

Format of crontab file

A cron tab entry has 5 fields. These are: minute, hour, day of month, month and day of week. For example:

*/30 * * * 1,3

The above entry will run every 30 minutes on Mondays and Wednesdays

Running Cron Jobs at random intervals

To run a Cron Job at random intervals we need to prefix the Cron Job command with the sleep bash command. For example to run a Cron Job every 5 to 7 hours at random intervals, we can use the following:

0 */5 * * * sleep $[RANDOM\%120]m; path-to-script

In the above code, the ’%’ has to be escaped. The code: $[RANDOM%120] generates a random number between 0 and 120. This is the number of minutes that script will pause. The script will run at the times: 05:00+(rand(0, 120)), 10:00+(rand(0, 120)), 15:00+(rand(0, 120)), 20:00+(rand(0, 120)) and 00:00+(rand(0, 120))

Running cron jobs on first weekday of month

To run a cron job on the first weekday of every month for example first Wednesday of each month, the following cron job entry can be used:

0 0 * * Wed [ `date +\%d` -le 7 ] && command-to-run

The above cron job will be run on first Wednesday of each month

Database Management - Quick Tips

Wed, 03 Apr 2019 00:00:00 GMT

Following are some quick tips related to Database Management:

Ignoring errors during MySQL import

While importing large database backups we may come across errors such as duplicate indexes. It would be useful to ignore such minor errors. The mysql command provides the —force switch which allows us to ignore the errors. For example:

sudo mysql -u {mysql_user} -p --force &lt; {sql_file}

List all databases for Postgresql

To list all PostgreSQL databases, we need to first switch to user postgres using the command su psql. Next we need to login to the PostgreSQL console using the command PSQL. After that run the command \list. This will list all databases. To run queries on the database fist run the command \connect database_name. Then run your query.

Restoring a database backup in Plesk

If your Plesk settings are messed up then it may be useful to restore the backup of the Plesk databases. Plesk automatically backups up the following databases every day: psa, horde and mysql. For example the psa database can be restored with following command: plesk db psa < psa.sql

To login to MySQL server as root privileges on a Plesk server we can use the following command: ** MYSQL_PWD=cat /etc/psa/.psa.shadow mysql -u admin**

Deleting a Mongo Db database

To delete a Mongo Db database called test, enter the command: mongo. This will take you to the Mongo Db shell. Then enter the commands:

use test;
db.runCommand( { dropDatabase: 1 } );

Create backup of Mongo Db database

To backup a Mongo Db database, we can use the command: mongodump —archive=test.gz —gzip —db test. It will create a gzip backup of the test database

Troubleshooting problems with MySQL server using MySQL Tuner

The MySQL tuner is a Perl script that checks a MySQL server and gives suggestions on how to improve its performance The script checks MySQL server log files and configuration and suggests optimal values for server configuration variables such as query_cache_size.

MySQL server out of resources

If you get following error on your MySQL/MariaDB server: General error: 23 Out of resources, then a possible solution is to increase the open_file_limit mysql variable. On Centos 7, this can be done by first creating the folder: /etc/systemd/system/mariadb.service.d/, then create a new file limits.conf with the following contents:

[Service]
LimitNOFILE=20000

After that restart the MySQL/MariaDB service with the command:

systemctl daemon-reload
systemctl restart mariadb.service

Sorting a MySQL string column as a number

To sort a MySQL string column containing a number we can use the SUBSTRING and CAST functions. The SUBSTRING function should extract the numbers inside the string. The CAST function will convert the extracted string to a number.

MySQL user password not being set

If you have problems with setting the user password on MariaDB or MySQL, for example you set the user password and are not able to login with the new password, then the problem may be caused because the database user is configured to use the wrong authentication plugin. For example unix_socket plugin instead of the default authentication scheme.

The solution is to set the plugin field in the user table of the MySQL database to empty value. This will cause the MySQL/MariaDB server to authenticate the database user with the default authentication scheme.

Removing ONLY_FULL_GROUP_BY sql mode

In MySQL 5.7.5 the ONLY_FULL_GROUP_BY SQL mode was introduced. In this mode, Non-deterministic grouping queries are rejected.

Applications that use GROUP BY clause in SQL queries may start giving errors after an upgrade to the MySQL server. To allow the applications to continue working, we need to remove the ONLY_FULL_GROUP_BY option from the SQL mode variable. This can be done by setting the SQL mode variable in /etc/mysql/mysql.conf.d/mysqld.cnf without the ONLY_FULL_GROUP_BY option. After that the MySQL server should be restarted. To get the current value of the SQL mode variable go to the variables tab in PhpMyAdmin and copy the value of SQL mode variable.

Batch rename MySQL database tables

Sometimes we need to rename our MySQL database tables in bulk. For example we may decide to add or change the table prefix on all tables within a database. Database management tools such as PhpMyAdmin allow us to rename tables with a few clicks, but most tools do not allow batch renaming.

The article In MySQL, how do I batch rename tables within a database? suggests that to rename several MySQL tables at once, we can run a SQL query that fetches the list of all database tables from the information_schema.tables table. The SQL query can generate rename table statements. We can then export these rename table statements to a file. We can further process the table names using the find/replace feature of standard text editors. Next we can run the SQL queries for renaming the database tables.

For example the following SQL query will append the text “_old” to all tables in the “db” database. The query will not actually rename any tables. It will simply generate rename table statements. In order to restrict the renaming to certain tables, we can update the WHERE condition.

SELECT concat ('rename table ', table_name, ' to ', table_name, '_old;') FROM information_schema.tables WHERE table_schema='db'

MySQL query to calculate running balance

It is possible to calculate a running balance using only MySQL. A running balance is similar to the last column of a bank statement. Since the running balance depends on the balance of the previous row, we need to use a temporary table. The following three SQL queries can be used to calculate a running balance:

1. create TEMPORARY table tbl_temp (select * from tbl);
2. update tbl t1 set t1.Bal=(SELECT SUM(t2.deposited_amount-t2.issued_amount)
FROM tbl_temp t2 WHERE t2.id<=t1.id);
3. select * from tbl;

Profiling MySQL queries using PhpMyAdmin

To see which SQL queries are currently running on your MySQL server, enter the following SQL commands:

SET GLOBAL general_log = 'ON';
SET GLOBAL log_output = 'TABLE';

This will log the SQL queries to the table: “general_log” in MySQL database.

To see which slow SQL queries are currently running on your MySQL server, enter the following SQL commands:

SET GLOBAL slow_query_log = 'ON';
SET GLOBAL log_queries_not_using_indexes = 'ON';
SET GLOBAL log_output = 'TABLE';

This will log all the slow SQL queries to the table: “slow_log” in MySQL database. It will also log queries that are not using indexes.

HTML, JavaScript and CSS - Quick Tips

Wed, 03 Apr 2019 00:00:00 GMT

Following are some quick tips related to HTML, JavaScript and CSS:

Free Font for Urdu language

A free font for Urdu language is provided by a Pakistani Organization called Center for Language Engineering (CLE). The organization conducts research and development in linguistic and computational aspects of languages, specifically of Pakistan and developing Asia

How to use CCS rules specific to FireFox

The CSS statement:

@-moz-document url-prefix() {
  h1 {color:red}
}

allows the CSS rule: h1 {color:red} to be applied only to Firefox browser

Using ems instead of Pixels for Typography

EM is a unit for font sizes. It is based on the font size of the container element. For example if an element has a font-size set to 2em and the font-size of the container element is set to 20px, then the font-size of the target element will be 40px

Replacing a string using Javascript replace function

The replace function can be used to replace a substring with another string using regular expressions. For example: str.replace(/substring/g, ‘New Substring’); will replace all occurrences of ‘substring’ with ‘New Substring’

Base64 Encoding of utf-8 encoded string using Javascript

The btoa function encodes an ASCII encoded string using Base64 encoding scheme. If the string is utf-8 encoded then the btoa function will not work correctly. The solution is to first url encode the string using encodeURIComponent and then call window.btoa function.

Passing parameters to Jquery event handler function

To pass a parameter to Jquery event handler, the parameter should be given as first argument to event function. The argument will be passed on as a property of event.data. For example:

jQuery(".selector").change({parameter: value}, function(event){
    user_function(event.data.parameter);
});

Fetching HTML elements using JavaScript

Fetching parent node element of a HTML node. We can fetch the parent node of a HTML element using following code:

var x = document.getElementById("myLI").parentNode.nodeName;

Fetching HTML elements using CSS class names. We can fetch HTML elements from CSS classes using following JavaScript code:

var x = document.getElementsByClassName("example");

Fetching html elements using CSS selectors with Javascript. The following Javascript code fetches html elements using CSS selectors:

var x = document.querySelectorAll(".example");

Get current unix timestamp using Javascript

The Javascript function getTime returns the number of milliseconds since Jan 1st, 1970, which is called the unix timestamp

Adding tilt effect to images using tilt.js

Tilt.js is a library that allows adding tilt effect to HTML elements. For example div and image tags. To add a tilt affect to an image, the following code may be used:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <title>Tilt.js - DEMO</title>
    <script src="js/jquery.slim.min.js"></script>
    <script src="js/tilt.jquery.min.js"></script>
    <script>
      $(document).ready(function () {
          var tilt = $('.js-tilt').tilt();
          var tiltOutput = $('.js-tilt-output').tilt();
          tiltOutput.on('change', function (e, transforms) {
              var output = $(this).closest('.js-parent').find('.js-output');
              $('<li><strong>X: </strong>' + transforms.percentageX + ' | <strong>Y: </strong>' + transforms.percentageY + '</li>').prependTo(output);
          });

          $('.js-destroy').on('click', function () {
              var element = $(this).closest('.js-parent').find('.js-tilt');
              element.tilt.destroy.call(element);
          });

          $('.js-enable').on('click', function () {
              var element = $(this).closest('.js-parent').find('.js-tilt');
              element.tilt();
          });
      });
    </script>
  </head>
  <body>
    <main>
      <a target="_blank" href="[some-url]">
	<img src="[image-url]" class="js-tilt" data-tilt-speed="1000" data-tilt-max="20" data-tilt-scale="1.2" data-tilt-perspective="250" width="961px"></a>
    </main>
  </body>
</html>

Scrolling to specific position in page

To scroll the page to a particular position, the following code may be used:

var temp_arr = location.href.split('#');
var div_id   = temp_arr[1];
var position = $('#' + div_id).position();
window.scrollTo(0, (position.top -100));

The above code causes the current page to scroll to a certain div. The id of this div is given by the URL hash fragment.

The code uses the JQuery position function to get the left and top co-ordinates of the div tag. The window.scrollTo function is used to scroll 100px above this div tag, so it can be viewed properly.

Preloading font files

In order to use custom fonts that are not supported by the browser, we need to load a font file. Sometimes the font file may take a while to fetch, because of which the website text is displayed with default font. This usually happens with slow internet connections.

This can be a problem for websites that need to display text in a foreign language such as Urdu or Arabic. To prevent this problem, we can preload the font file. This allows the browser to load the font file before other content. It increases the chances of displaying text with correct font.

If the text is still not displayed with correct font, then we can use the Font API to load the font files. The Font API provides a JavaScript interface for loading font files. It allows monitoring loading of files. It can be used to hide text until a font file has completed loading.

The article Preloading @font-face fonts? discusses how to preload font files. The following code may be used to preload font files:

<link rel="preload" href="/ui/font/NafeesWeb.ttf" as="font" crossorigin="anonymous" />
<style>
@font-face {
  font-display: auto;
  font-family: NafeesWeb;
  src: local(NafeesWeb),url(/ui/font/NafeesWeb.ttf);
}

Linux Server Administration - Quick Tips

Wed, 03 Apr 2019 00:00:00 GMT

Following are some quick tips related to Linux Server Administration:

Recover deleted files on ext3/ext4 file system using Photorec

Photorec is part of testdisk package. To install testdisk package on Debian use apt-get install testdisk. To run photorec enter the command photorec.

Force grub2 to not use UUID

When restoring backup of virtual machine, the backup process may create a new disk with a new UUID. In such cases it may be useful to update grub2 so it uses the disk physical path instead of the disk UUID.

The server may not reboot if the UUID of the new disk is different from the UUID of the old disk. One possible solution is to ensure that /etc/fstab file uses physical paths instead of UUID. Another solution is to update grub configuration so it uses physical device paths e.g /dev/xvda1 instead of UUID of disk.

How to Backup and Restore LXC containers

It describes how to backup and restore LXC containers. One option for backing up a LXC container is to create a tar acrhive using the commands:

lxc-stop -n ContainerName;
cd /var/lib/lxc/ContainerName/
tar --numeric-owner -czvf container_fs.tar.gz ./*

The backup of the container may be restored using the command:

rsync -avh container_fs.tar.gz user@server_name:/var/lib/lxc/
tar --numeric-owner -xzvf container_fs.tar.gz ./*

Installing memcached for Php 7

Memcached is an extension for Php that allows accessing the Memcached daemon from Php. A similar extension called Memcache is also available but it is out dated and does not support Php 7.

To install Memcached for Php 7 we need to first fetch the source code from GitHub by cloning the php7 branch. Then we need to compile it from source using the following commands:

git clone -b php7 https://github.com/php-memcached-dev/php-memcached.git
phpize
./configure
make
make install

NOTE:. The Php memcached extension has now been updated to support Php 7.x. To install the Php memcached extension the following command can be used: pear install memcached.

Sending email through Gmail server using mailx command

Mailx command can be used to send email through the gmail smtp server. For example the following command can be used:

mailx -s "Memory usage on server is more than 70%. Please check!." -r "reply_email" -S smtp="smtp.gmail.com:587" -S  \
smtp-use-starttls -S smtp-auth=login -S smtp-auth-user="gmail_address" -S smtp-auth-password="gmail_password" -S \
ssl-verify=ignore recepiant_email &lt; /tmp/mail_body.txt

In the above command the “gmail_address” needs to allow access from insecure devices. This article explains how to do this.

List users currently logged in

To list the currently logged in users, we can use the command: users.

Upgrading Rocket Chat to latest version

To upgrade Rocket Chat to the latest version we need to first download the latest version of Rocket Chat using the command:

curl -L https://rocket.chat/releases/latest/download -o rocket.chat.tgz

After that we need to unzip the file. Then we need to remove the content of the current Rocket Chat folder with the contents of the unzipped file. After that we need to change directory to programs/server and run: npm install. If the command fails then we may need to install the node-pre-gyp package using the command: npm install node-pre-gyp. After that run the previous command. This should upgrade the Rocket Chat to the latest version.

Problem with copying large files to USB

If you are trying to copy a large file to a disk that has been formatted with the FAT32 file system or similar, then you might get the error: “Error splicing file: File too large”. The error can be corrected by formatting the disk with the ext4 file system.

Migrating Trello issues to Redmine

We can use this script to migrate Trello issues to Redmine: https://github.com/inequation/trello2redmine To use the script, we first need to configure it with the Redmine url, project id, default user and a few other parameters. We can run the script with the command:

python trello2redmine.py -c

Checking Email server for problems

The MxToolBox is an online tool that allows checking Email server for problems such as missing PTR records, open relays etc. Its a useful tool for troubleshooting network related problems.

Debugging programs on Linux using strace

Strace is a system tool that is used to debug programs on Linux based operating systems. It uses a feature of the Linux kernel known as ptrace.

Strace as its name suggests is used to trace operating system calls. It can be used to find out if a program is getting stuck because of a missing file or because of shared memory problems. For example it can show the id of the semaphore that is causing the program to fail. This semaphore can then be removed using the ipcrm command. The ipcrm command allows removing System V inter-process communication (IPC) objects and associated data structures from the system.

Increasing number of concurrent connections for Proftp

To set the number of concurrent connections for Proftp, we need to set the following configuration directives in the Proftp configuration file: MaxInstances, MaxClients, MaxClientsPerHost, MaxClientsPerUser, MaxHostsPerUser. These directive are set when Proftp is running in standalone mode.

If Proftp is running in inet mode, then per_source and cps attributes need to be set in the xinetd configuration file.

Using PSCP command for copying files and directories

PSCP is a command that allows uploading files and folders to a remote server. Its benefit is that it does not require rsync to be installed on the remote server. The PSCP command can be used for copying data to a shared hosting platform which does not support rsync and does not provide shell access. It can also be used to transfer data between Windows and Linux servers.

The following command can be used to recursively copy a folder to a remote destination:

pscp -r {source_folder} {user_name}@{remote_server}:{remote_folder_path}

Finding name server records for domains using Dig

Dig is a useful command line tool which allows retrieving DNS records for a domain.

Authoritative name servers are servers that provide DNS information about a domain. An Authoritative name server may be a primary or secondary name server. A primary name server contains definitive and reliable information about a domain. A secondary name server is a slave name server which contains replica of dns records on a primary name server.

To fetch the list of name server records for a domain we can use the command: dig NS {domain_name}. If we add the +short flag, then a summary of the information is returned. Other types of DNS records may be fetched similarly.

Plesk Repair Utility

The Plesk repair command is used to automatically diagnose and resolve issues with Plesk. To use the command, login to your Plesk server over SSH and enter the command: plesk repair ASPECT [OPTION]. ASPECT indicates the service to repair while OPTION indicates how the repair should be performed.

ASPECT can have several values such as web, mysql, mail, all etc. OPTION can have the values: Interactive mode, Diagnostic mode or Repair mode. The default option which is Interactive mode, prompts the user to take action when an issue is found. In Diagnostic mode, the issue is indicated but no action is taken. In Repair mode, the issue is automatically resolved.

Checking hard disk for bad blocks

A hard disk bad block is a section of the hard disk that cannot be read from or written to. To detect bad blocks we can use the badblocks command. For example to scan the hard drive /dev/sda for bad blocks, we can run the command: badblocks -v /dev/sda.

Listing files and directories sorted by size

To find out which files and directories under a given directory are taking up the most space, the following command can be used:

du -a /home | sort -n -r | head -n 5

Setting JAVA_HOME in Debian

The JAVA_HOME environment variable is set to the path of the JDK installation. On Debian 9 the location of the JDK is /usr/lib/jvm/java-11-openjdk-amd64 for JDK version 11. The variable should be set in /home/[user]/.bashrc file, where [user] is the system user account used to run Java applications. For example add following to /home/user/.bashrc file:

export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-amd64"
export PATH=$JAVA_HOME/bin:$PATH

Linux Command Line Tools - Quick Tips

Wed, 03 Apr 2019 00:00:00 GMT

Following are some quick tips related to Linux Command Line Tools:

Using SED tool for replacing text in files

The following script can be used for updating all occurrences of a text inside files:

find folder_name -type f -print0 | xargs -0 sed -i 's/current_text/new_text/g'

Using SED tool for removing line that starts with some text in all files

The following script can be used for updating all occurrences of a text inside files:

find . -type f -exec sed -i '/^category:/ d' {} +

Using Rsync fore recursive file copy

To use Rsync for copying all mp3 files recursively from one folder to a single folder, use following command: rsync -avz ./..mp3 destination_folder/. The command should be issued from the source folder

Viewing number of files in directory

Use following command for fetching number of files in directory: ls | wc -l

Convert pdf files to text files

The pdftotext utility which is part of poplar-utils package allows converting pdf files to text files

Automate scp command using sshpass

To use the scp command non-interactively, for example in a script, we can use the sshpass utility. The following command can be used to copy a file to a remote server:

sshpass -p '{password}' scp {source_file_name} {remote_user}@{remote_server}:{remote_server_path}

Delete downloaded file using Wget

The delete-after command line option of wget is used to delete the downloaded file

Fetching website resource over https using wget

To fetch a website resource over https, we can use wget command. If the website’s https certificate is not trusted by wget, then we can use the —no-check-certificate switch with wget. For example:

wget https://example.com/path_to_resource --no-check-certificate

Monitoring user activity with acct package

The acct package provides tools for monitoring user activity. For Centos/Fedora this package is called psacct. For Debian based systems it is called acct One of the tools provided by the acct package is called sa. It prints a summary of the command executed by users Another tool called ac lists the total time for which users have been logged in

Split large files using split command in Linux

To split a large file into smaller files, we can use the split command that is part of most Linux distributions. For example the following command splits a file into smaller files of size 5 Mb: split large_file_name -b 5M; To combine the file parts back to the original file, we need to use the command: cat x > original_file_name*

Using Rsync over custom SSH port

Rsync can be used with custom SSH ports. For example to copy data from a remote server that is running SSH server over port 3000, the following command can be used:

rsync -avze "ssh -p 3000"  {user}@{server}:{remote_folder} {local_folder}

Setting file permissions with rsync

If we need to upload files to a remote server using rsync, and the files need to have certain ownership, then the chown option of rsync can be used. For example the following rsync command uploads the contents of the current folder to a remote server and sets the owner and group of the uploaded data to www-data:

rsync -avz * {remote_user}@{remote_server}/{remote_server_path} --chown=www-data:www-data

Rsync over SSH with key authentication

The following rsync command can be used to copy files on a remote server using SSH key authentication:

rsync --progress -avz -e "ssh -i {key_file_location}" \
remote-user@remote-server:remote-folder local-folder

Deleting files older than 1 year using Linux command line tools

If you want to recursively delete all files in a folder that were modified over 1 year ago, then the following command can be used:

find folder_path -type f -mtime +365 -ls -exec rm -f -- {} \;

Deleting empty folders recursively using Linux command line tools

If you want to recursively delete all folders within a given folder that are empty, then the following command may be used:

find . -type d -empty -delete

The following command can be used to recursively list all empty folders within a given folder:

find . -type d -empty

Renaming multiple files in folder with BASH

We often have a need to rename all files at once. For example we may need to add a certain text to all files within a folder. We can do this using a programming language. However the Bash programming language allows us to do this very easily using the rename command.

For example to add the text “_test” before the file extension to all mp3 files in a directory, we can use the following command:

rename -n 's/(.*)\.mp3/$1_test.mp3/' *.mp3.

This was suggested in an answer to the question Add text to end of the filename but before extension on StackOverflow.

Recursively change folder ownership

If you want to change the permissions of all folders under current directory to 755

find . -type d -exec chmod 755 {} \;

Appending text to end of file with Bash

To append text to the end of a file in Linux:

echo "Some new text" >> /tmp/file_name.txt

PHP and Python - Quick Tips

Wed, 03 Apr 2019 00:00:00 GMT

Following are some quick tips related to PHP:

Fixing timeout problems with PHP scripts on Plesk

In order to allow long running PHP scripts to run on a Plesk server, we need to increase the max_execution time value in the PHP configuration settings. If the Plesk is running PHP-FPM or PHP Fast CGI, then the following directives should be added to the file /etc/httpd/conf.d/fcgid.conf:

FcgidIOTimeout 9000
FcgidBusyTimeout 3600

The above options can also be added to the Apache settings for the domain

PHP server APIs

PHP may be run as an Apache module. This configuration is known as mod_php. It may also be run as a FPM server or a Fast CGI server. In the Fast CGI configuration, the PHP binary is accessed by the web server using the Fast CGI protocol. In the PHP FPM configuration, the PHP is managed by a Fast CGI server called PHP FPM. The web server communicates with this PHP FPM server over a TCP socket.

Fetching Gzip content with PHP Curl

To fetch urls such as RSS Feeds that have been compressed with gzip encoding the following curl option can be used:

curl_setopt($ch,CURLOPT_ENCODING , "gzip");

Using Usort function for sorting Arrays

The PHP usort function allows sorting an array of values using a custom callback function. The usort function takes a comparator callback function as an argument. This function takes two arguments which are the two values to compare. The strcmp function can be used inside the comparator function for comparing string values.

PHP isset function

The PHP function isset returns false for null values. It should not be used to check if array key exists, since some array keys may have empty value. Its better to use array_key_exists function instead.

Generating random strings with array_rand and shuffle functions

The PHP functions array_rand and shuffle functions can be used to generate random strings.

What is the difference between a PHP interpreter and a PHP handler?

A PHP Handler is an Apache module that is used by Apache to communicate with the PHP Interpreter. It is basically used by Apache to handle requests for PHP files. There are different types of PHP Handlers.

For example mod_php is an Apache module for PHP. mod_fcgid allows communication with the php_fpm interpreter over a TCP or Unix socket. The PHP Interpreter is an application that executes PHP code one line at a time. The output of the PHP Interpreter is sent to the PHP Handler. This output may be processed by other Apache modules such as mod_deflate or mod_security. The final output is sent out of the TCP socket that Apache listens on.

SuPHP is an Apache module that allows executing PHP scripts with the permissions of the script owner. Normally PHP scripts are executed with the permissions of the PHP interpreter. SuPHP allows the PHP files to have custom owner and permissions. The PHP file is then executed by the same owner that owns the PHP file. Its useful in shared hosting environments where PHP files belonging to different users need to be executed by the same PHP interpreter.

Passing command line arguments to Python script

Command line arguments can be passed to a Python script. For example the parameters passed in the following command:

python script_name argument1 argument2 argument3

can be read with the sys module. The sys module provides the command line arguments. To use the sys module, we have to use the command:

import sys

The command line arguments can be used as: sys.argv[0], sys.argv[1], sys.argv[2]. The number of command line arguments can be read with: len(sys.argv).

PHP 7 error handling

PHP 7 uses a different approach for handling errors. It uses a different Error hierarchy then PHP 5. The base error class in PHP 7 is Throwable. So if we want to catch all exceptions that can be raised by a given code, then we should enclose the code in try/catch block. The catch block should use the Throwable class. For example:

try {
// some php code
}
catch (Throwable $e) {

}

Executing PHP code using eval

The PHP eval function allows executing arbitrary PHP code. It takes as argument a string containing PHP code it returns null or the value returned by the PHP code.

Generate URL query strings using http_build_query

The http_build_query function allows generating valid URL query strings from simple or complex arrays including nested arrays.

Version Control - Quick Tips

Wed, 03 Apr 2019 00:00:00 GMT

Following are some quick tips related to Version Control:

Changing a remote GIT repository URL

The following command can be used to set the URL of a remote git repository: git remote set-url origin {remote-repo-url}

Error in pushing changes to remote git repository

If we have a git repository with large files, and we need to push changes to a remote repository, then we might get the following error:

Counting objects: 9554, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (5064/5064), done.
Writing objects: 100% (9554/9554), 2.79 GiB | 694.00 KiB/s, done.
Total 9554 (delta 4382), reused 9554 (delta 4382)
error: RPC failed; result=22, HTTP code = 502
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

To correct this error we need to increase the size of the git configuration variable http.postBuffer. We can do this with the command:

git config --global http.postBuffer 157286400

Updating last commit without changing annotated tags

If you need to make modifications to the last commit of a Git Repository, without making changes to the annotated tags, then the following script may be used. The script also updates the remote repository.

# add changes to staging area
git add .
# changes in the staging area are added to the last commit. last commit message can also be changed
git commit --amend
# the annotated tag called "v1.0.0" is deleted. this should be the most recent annotated tag
git tag -d v1.0.0
# the most recent annotated tag is deleted from the remote server
git push origin --delete v1.0.0
# a new annotated tag is added, pointing to the last commit
git tag -a v1.0.0 -m "initial commit"
# the last commit is pushed to the remote repository
git push origin master -f
# all annotated tags are pushed to the remote repository
git push --tags

Storing password in SVN

To store password for SVN commit commands, set store-passwords and store-plaintext-passwords to yes in /home/home_folder_name/.subversion/servers. To set the user name in the commit command, append —username {your-user-name} to the SVN commit command.

Update commit message history

To rename previous commit messages, enter the command:

git rebase -i HEAD~n

In the above command n is the number of commits back in history since the last commit. The command will cause the command line editor to open with the last n commit messages. Each commit message is prefix with the text: pick. Replace this with reword for the commit message that needs to be updated.

Next save the changes and exit the editor. This will cause a new editor window to open allowing editing of a commit message. After changing the commit message, save the changes and exit the editor. The editor will open up again allowing editing of the next commit message. Do this for each commit message that needs to be updated.

Editing particular commit

To edit files belonging to a particular commit, enter the command:

git rebase --interactive 'SHA-1^'

SHA-1 is the hash for the particular commit. It can be found using the command: git log —oneline

This will open a command line editor which will list all commits made after the particular commit. Change the text “pick” to “edit” besides the particular commit. Save the changes and exit the editor.

This will switch the HEAD to the given commit and will copy the commit files to the current working tree. The current working tree will be just like it was when the commit was made. Next make the necessary changes to the files and commit the changes using the command:

 git commit --all --amend --no-edit

After that enter the command:

git rebase --continue

This will make the changes to the particular commit and will switch the HEAD back to the master branch. After that the changes can be pushed to the configured remote git repository using the command:

git push -u origin master --force

Removing large files from all commits

To remove large files from all commits in a git repository, the git-filter-repo tool can be used. To install the tool follow this install guide.

Next run the following command to remove the directory dir1/dir2 from all commits:

git-filter-repo --path "dir1/dir2" --invert-paths

The path option specifies the files or folders to include. The invert-paths option inverts the paths so the end result is that all files and folder except those given by the path option are included. In this way files and folders can be removed.

The git-filter-repo supports many other options including removing objects larger than given size and editing commit messages.

Tagging specific commit

To tag a specific commit, the following command can be used:

git tag -a tag-version commit-checksum -m "commit messsage"

The above command adds an annotated tag named “tag-version”. The commit-checksum is the checksum of the commit. It can be found using the command:

git log --oneline

git-filter-repo --path "dir1/dir2" --invert-paths

The git-filter-repo supports many other options including removing objects larger than given size and editing commit messages.

Developers Site

Mon, 18 Mar 2019 00:00:00 GMT

Introduction

The Developers Site project is a simple website that displays articles. It is suitable for developers who need a platform for publishing their work.

Development of the website

The website was developed by following the Lean Startup principles. The book Understanding Software was an invaluable guide in the development of the website.

Features

The website allows publishing articles
Table of contents are auto generated for articles with headings
Comments can be posted to articles
Contact form can be used to send a message to the website owner
The website has a search box for searching articles
List of recent articles and categories is auto generated
The source code is modular and easy to extend with new features
The website front-end is based on Twitter Bootstrap and JQuery
The website front-end is valid HTML5 and has been tested with the validator.nu tool
The article structure is based on Semantic Elements and is SEO friendly
The website is mobile friendly
The website has good browser response time
Google Analytics tracking code can be easily added for keeping track of website visitors
The website provides scripts for generating Site Maps and for testing the website pages for broken links and HTML 5 validation errors

Limitations

The website does not have a back-end. Article content has to be added directly to database using a database management tool such as PhpMyAdmin

Requirements

The “Developers Site” is based on PHP and MySQL. It uses the Twitter Bootstrap and JQuery Frontend libraries. The website uses the Mail and Mail Mime pear packages for sending email. These packages should be installed on your server.

Installing the website on your own server requires PHP 7.2 and above. The website source code is based on the Pak Php Framework. The source code is fully commented and compliant with the PSR-2 coding guidelines. The source code for the website is modular and easy to extend with new features.

Site Map Generation

A Site Map for the website pages can be auto generated by running the command:

php index.php  --application="pakjiddat" --action="Generate Site Map";

This creates a file sitemap.txt in the data folder. The script also saves all URLs to the pakphp_test_data, which allows each URL to be checked for broken links and HTML 5 compatibility.

Markdown file Generation

The website content stored in database can be exported as markdown files by running the command:

php index.php --applicaiton="pakjiddat" --action="Generate Markdown";

This creates markdown files in the data/markdown folder.

Testing

All pages on the “Developers Site” website can checked for broken links and validated using the validator.nu tool. To test the website pages, first enter the list of URLs to test in the database table: pakphp_test_data. The table may be auto populated by setting the variable: save_ui_test_data to true in pakjiddat/config/Test.php file. Alternately the Site Map script may be run, which saves all Site Map urls to the database table. Next run the command:

php index.php  --application="pakjiddat" --action="Unit Test";

This will start the testing of each page listed in the database table.

Installation

The following steps can be used to install the “Developers Site” project on your own server:

Download the source code from GitHub
Move the source code to the document root of a virtual host
Create a database and import the contents of the file: pakjiddat/data/developers-site.sql to the database. Note down the credentials used for connecting to the database
Enter the database credentials in the file pakjiddat/config/RequiredObjects.php
In the file: pakjiddat/Config.php, on line 41 enter the domain names that will be used to access the website
Customize the following variables in the file: pakjiddat/config/General.php. $config[‘app_name’], $config[‘dev_mode’] and $config[‘site_url’]
Customize the variables in the file: pakjiddat/config/Custom.php. The comments explain what each variable is used for
Set the $config[‘pear_folder_path’] variable in the file: pakjiddat/config/Path.php. The variable should be set to the path of the pear installation.
Visit the website in a browser
Enter the website articles in the database table: home_content. The article should start showing on the website
The layout and text of the website can be edited in html files inside the folder: pakjiddat/ui/html. For example to edit the website header, the file: pakjiddat/ui/html/base/header.html needs to be edited

Islam Companion Web API

Mon, 18 Mar 2019 00:00:00 GMT

Introduction

The ”Islam Companion Web API” project is a RESTFul API (Application Programming Interface) that allows users to add Holy Quran and Hadith data to their applications.

It provides Holy Quran translation in 42 languages. Following languages are supported: Amharic, Arabic, Bosnian, Bengali, Bulgarian, Amazigh, Czech, German, Divehi, Spanish, English, Persian, French, Hindi, Hausa, Indonesian, Italian, Japanese, Korean, Kurdish, Malayalam, Malay, Dutch, Norwegian, Portuguese, Polish, Russian, Romanian, Swedish, Somali, Sindhi, Albanian, Swahili, Turkish, Tajik, Tamil, Tatar, Thai, Uzbek, Urdu, Uyghur and Chinese.

Hadith translation is provided in Urdu, English and Arabic languages.

The goal of the Islam Companion Web API is to help users develop applications that promote knowledge about Islam.

Features

The Islam Companion Web API has the following features:

It provides translations of Holy Quran in 42 languages
It provides translations of Hadith in Urdu, English and Arabic languages
It provides 8 API functions for fetching Quranic data
It provides 7 API functions for fetching Hadith data
The source code is available under GPL License
The source code is well commented and easy to update

Requirements

The Islam Companion Web API requires PHP >= 7.2. It also requires MySQL server >= 5.6.

Installation

The following steps can be used to install the “Islam Companion Web API” project on your own server:

Download the source code from GitHub
Move the source code to the document root of a virtual host
Download the contents of the database from: here
Extract the downloaded file
Create a database and import the contents of the SQL file to the database. Note down the credentials used for connecting to the database
Enter the database credentials in the file api/config/RequiredObjects.php
In the file: api/Config.php, on line 37 enter the domain names that will be used to access the API
Customize the following variables in the file: api/config/General.php. $config[‘app_name’], $config[‘dev_mode’] and $config[‘site_url’]

Download data

Click here to download the Hadith database in .sql format for MySQL server.
Click here to download the Hadith database in .db format for SQLite server.
Click here to download the Quranic database in .db format for SQLite server.

Frequently asked questions

What is the Islam Companion Web API. It is a RESTFul API for accessing Holy Quran and Hadith data
What can I do with the Islam Companion Web API. You can develop web based applications that present Holy Quran and Hadith data to the user. The Islam Companion API can work as the back-end of your application.
Which languages are supported by the Islam Companion Web API. Click Here
What functions are provided by the Islam Companion Api. Please read the documentation.
From where does the API get its data. The Islam Companion API uses Holy Quran translations from http://tanzil.net/trans/. It uses Hadith data from http://hadithcollection.com/ and https://www.quranurdu.com/Ahadith/.
How do I use the API. To use the API, you have to make HTTP POST request to your server. Please see following sample code in PHP language.

Sample Code

<?php

$data = array(
    'language' => 'English',
    'narrator' => 'Mohammed Marmaduke William Pickthall'
);

// Prepare new cURL resource
$ch = curl_init('https://your-host-name/api/get_random_verses');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

// Submit the POST request
$result = curl_exec($ch);

// Close cURL session handle
curl_close($ch);

// json decode the response
$result = json_decode($result, true);

// Print the result
print_r($result);
?>

Usage

The Islam Companion Web API supports the following functions. All function response values are in JSON format. To test the API, use the following url: https://your-host-name/api/ for making API requests. All API requests must be of type HTTP POST and should include the required API parameters.

API calls for fetching Holy Quran data

API calls for fetching Hadith data

Islam Companion Website

Mon, 18 Mar 2019 00:00:00 GMT

Introduction

The Islam Companion website project is a website that allows users to read Holy Quran and Hadith text. It also allows users to subscribe to Holy Quran and Hadith by email.

The goal of the Islam Companion website is to provide a basic website that displays Holy Quran and Hadith data. The user may customize this website according to their requirements.

Features

The Islam Companion website has the following features:

It has a reader that displays Holy Quran verses
It has a reader that displays Hadith text
It allows users to get Quranic verses and Hadith by email
It allows downloading Hadith database in .sql format
It allows users of the website to send a message to the website administrator
The source code is available under GPL License
The source code is well commented, modular and easy to update

Limitations

The Islam Companion website has the following limitations:

The Islam Companion website was designed using JavaScript modules which are not supported by older browsers

Requirements

The Islam Companion website requires PHP >= 7.2. It also requires MySQL server >= 5.6. For sending emails, the website requires the Mail and Mail Mime pear packages.

Installation

The following steps can be used to install the “Islam Companion” project on your own server:

Download the source code from GitHub
Move the source code to the document root of a virtual host
Download the contents of the database from: here
Extract the downloaded file
Create a database and import the contents of the SQL file to the database. Note down the credentials used for connecting to the database
Enter the database credentials in the file islamcompanion/config/RequiredObjects.php
In the file: islamcompanion/Config.php, on line 32 enter the domain names that will be used to access the website
Customize the following variables in the file: islamcompanion/config/General.php. $config[‘app_name’], $config[‘dev_mode’] and $config[‘site_url’]
Customize the variables in the file: islamcompanion/config/Custom.php. The comments explain what each variable is used for
Set the $config[‘pear_folder_path’] variable in the file: islamcompanion/config/Path.php. The variable should be set to the path of the pear installation.
Visit the website in a browser
The layout and text of the website can be edited in HTML files inside the folder: islamcompanion/ui/html. For example to edit the website header, the file: islamcompanion/ui/html/base/header.html needs to be edited

Responsive website design with Element Queries

Mon, 18 Mar 2019 00:00:00 GMT

Element Queries allow changing the size of a HTML element based on the style of other elements. Element queries are similar to media queries, but are based on element size instead of browser or view-port size.

Element Queries are not supported by browsers natively like media queries. Two libraries that implement element queries are: EQCSS and CSS Element-Queries. Both libraries are small sized JavaScript files, that parse contents of CSS files and run the element queries defined in the file.

The CSS Elements Query library has a feature called ResizeSensor. It allows JavaScript code to be run, when an element is resized. It can be used to change layout of pages that display dynamic content such as ads.

The Specification for CSS Element Queries describes how CSS Element Queries work

See the question Can media queries resize based on a div element instead of the screen? on Stack-overflow for more information about Element queries

YouTube Playlist

Mon, 18 Mar 2019 00:00:00 GMT

Introduction

The YouTube Playlist is derived from the MediaElement player. It allows YouTube videos to be displayed in a playlist.

Features

Allows playlist information to be loaded from XML file
Allows video quality to be set
Provides option for sharing video on social networks
The start and end time for each YouTube video can be specified in XML file. It allows unnecessary parts of the YouTube video to be excluded
Allows switching to next and previous tracks
Provides a loop option for playing a track repeatedly
Provides a shuffle option for playing a random track
Can be used under the terms and conditions of the Media Element Player license

Development of the player

The YouTube Playlist was developed by modifying the MediaElement Playlist plugin and the Quality plugin.

Minor changes to the MediaElement Player were also made. The article Mediaelements: Add a share button to video elements using jQuery was used to add Social Sharing buttons to the player. The player also uses the jQuery thumbnail scroller library for the scroll-able playlist.

Installation

Download the source code by running the command: git clone https://github.com/pakjiddat/youtube-playlist.git. Place the source code in a virtual host directory, so that the index.html file is served by a web server. Navigate to the index.html file in a browser. You should see this screen.

Generating Table of Contents

Sat, 16 Mar 2019 00:00:00 GMT

Introduction

Table of contents gives an overview of the article. It describes what the article contains. A quick glance at the table of contents gives a useful overview of the article. Most articles have a table of contents which contains links to the important parts of an article.

Basic Usage

To generate a table of contents using PHP, the following code can be used:

$toc = new TOC();
$toc_data = $toc->GenerateToc($article_text);

How the code works

In the above code, the $article_text is the article for which the table of contents needs to be generated. The $toc_data variable is an associative array containing two entries.

The toc_list entry is the table of contents formatted as an unordered HTML list. The updated_text is the updated article text. The headings in this updated text are marked with ids, so when the user clicks on a link in the table of contents, it scrolls to the article heading

The code for generating the table of contents does three things. First it extracts the headings from the article text using regular expressions. The headings are extracted recursively. Next the script generates an unordered HTML list from the extracted headings. After that the script updates the article text, adding ids to the HTML headings

Features

The script extracts all heading tags from h1-h6. It also adds ids to the article text. The table of contents is an unordered HTML list. Each item in the list is a link to a heading.

In the article text a heading level may be skipped. For example h5 headings can follow a h3 heading

Limitations

The headings in the article text must be nested. So after h1, there must be a lower level heading such as h2, h3, h4, h5, or h6.

Download PHP class for generating Table of Contents

Pak Php Framework

Mon, 11 Mar 2019 00:00:00 GMT

Introduction

The Pak Php Framework is a framework for developing PHP applications. It can be used to develop command line applications as well as browser applications based on Model View Controller (MVC) design pattern.

An application framework provides libraries and tools for developing applications. The Pak Php framework provides error handling, function validation, log handling, session handling, URL request handling based on MVC design pattern, URL routing, translation and testing. It also includes utility classes that provide commonly used features such as Template Engine, Database Abstraction, Error Handling, File and Folder management and more.

The utility classes and features provided by the Pak Php Framework are implemented as separate components that are easy to extend. All components of the Pak Php Framework are based on the principles of Separation of concerns and Don’t repeat yourself.

Third party frontend libraries such as Twitter Bootstrap, JQuery, W3.CSS etc can be integrated with the Pak Php Framework.

Requirements

The Pak Php Framework requires Php 7.2 and above. The code for the Pak Php Framework is fully commented and compliant with the PSR-2 coding guidelines. Parameter and return types are given for all methods.

The Pak Php Framework does not have any external dependencies apart from the XDebug PHP extension. It has its own Template Engine and Test Manager. It provides a simple framework for developing well tested applications.

Development of the website

The website was developed by following the Lean Startup principles. The book Understanding Software was an invaluable guide in the development of the framework.

The goal of the Pak Php Framework is to provide a light weight set of tools for developing professional, well tested web applications. Applications developed using the Pak Php Framework should be based on Lean Startup principles.

Application Structure

A single instance of the Pak Php Framework can support several applications. Each application must be in its own folder. The following shows the structure of a sample Pak Php Framework application:

├── config
│   ├── Custom.php
│   ├── General.php
│   ├── Path.php
│   ├── RequiredObjects.php
│   ├── Test.php
│   └── Callables.txt
├── Config.php
├── lib
├── test
│   └── results
│       ├── codecoverage
│       └── tracelogs
└── ui
    ├── css
    ├── html
    ├── images
    ├── js
    ├── pages

The config folder holds the application configuration files. Each file should contain a single method called GetConfig. This method should return an array containing configuration data, which overrides the default framework configuration. See the folder framework/config, for the default configuration files.

The Custom.php file can contain custom configuration. The Callables.txt file defines the URL routing information for the application.

The lib folder should contain the user defined library files. The ui folder contains the user interface related code. It should contains the sub folders shown in the above example. The css sub folder contains application CSS files, the images folder contains application image files, the js folder contains application JavaScript files. The html folder contains the HTML template files. The pages folder should hold files that derive from the \Framework\Application\Page base class.

Framework Structure

The Pak Php Framework has the following structure:

├── autoload.php
├── index.php
├── .htaccess
├── framework
│   ├── application
│   │   ├── Api.php
│   │   ├── Application.php
│   │   ├── CommandLine.php
│   │   ├── libraries
│   │   │   ├── ErrorHandling.php
│   │   │   ├── FunctionValidation.php
│   │   │   ├── LogHandling.php
│   │   │   ├── SessionHandling.php
│   │   │   ├── Translation.php
│   │   │   └── UrlRouting.php
│   │   ├── Page.php
│   │   └── Web.php
│   ├── config
│   │   ├── base
│   │   │   ├── GeneralConfig.php
│   │   │   ├── PathConfig.php
│   │   │   ├── RequiredObjectsConfig.php
│   │   │   └── TestConfig.php
│   │   ├── Config.php
│   │   ├── Initializer.php
│   │   └── Manager.php
│   ├── documentation
│   │   └── changelog.txt
│   ├── templateengine
│   │   ├── BasePage.php
│   │   ├── Generator.php
│   │   └── HeaderTags.php
│   ├── testmanager
│   │   ├── BlackBoxTesting.php
│   │   ├── CodeCoverageGenerator.php
│   │   ├── TestDataManager.php
│   │   ├── TestFunctionProcessor.php
│   │   ├── TestFunctionValidator.php
│   │   ├── TestResultsManager.php
│   │   ├── UiTesting.php
│   │   ├── UnitTestRunner.php
│   │   └── WhiteBoxTesting.php
│   ├── utilities
│   │   ├── Authentication.php
│   │   ├── autoload.php
│   │   ├── CacheManager.php
│   │   ├── commentmanager
│   │   ├── databasemanager
│   │   ├── documentation
│   │   │   └── changelog.txt
│   │   ├── Email.php
│   │   ├── Encryption.php
│   │   ├── errormanager
│   │   ├── examples
│   │   ├── Excel.php
│   │   ├── filesystem
│   │   ├── LogManager.php
│   │   ├── Profiler.php
│   │   ├── StringUtils.php
│   │   ├── TemplateUtils.php
│   │   └── UtilitiesFramework.php
│   └── vendors

Main entry point

The index.php file is the main entry point for the application. All URL requests are routed by the .htaccess file to the index.php file. The autoload.php file is used to autoload classes. All classes should have a name-space. The folder structure of a class should match the class name-space.

Application base classes

All Pak Php Framework applications are child classes of the Application base class, which is application/Application.php. Currently three types of applications are supported. API, Command Line and Web.

API applications extend the Api base class. Command Line applications extend the CommandLine base class. Web applications can extend either the Web class or the Page class. The Page class is derived from the Web class.

The main framework features which are URL Routing, Session Handling, Error Handling, Translation, Function Validation and Log Handling are implemented by classes in the application/libraries folder. Each feature is implemented separately from other features.

Configuration

Configuration is one of the main features of the Pak Php Framework. It allows class objects to be used without explicitly initializing the object. The user only has to mention the class once in config/RequiredObjects.php. For example:

$config['contactpage']['class_name'] = '\PakJiddat\Ui\Pages\Contact';

The class can then be used using the syntax: Config::GetComponent(“contactpage”);. The framework takes care of initializing the class object. If the object has already been created, then it is simply returned. All classes are autoloaded using PSR-4 autoloading standard.

Application data is stored in configuration files inside the folder: config/. See the user application structure for details. Configuration variables can be accessed using the syntax:

Config::$config["general"]["dev_mode"]

The above code returns the current development mode of the application. If it is false, then application is in production mode.

Url Routing

The Pak Php Framework provides URL Routing. The user application should contain the file config/Callables.txt. This file specifies the callbacks for commands as well as callbacks for handling URL requests. An entry in the file has two parts. The first part starts with url: or command:. The second part starts with callback:.

The application should define an entry for each URL or pattern of URLs. The first line in the entry is a regular expression that defines which URLs to handle. The next line defines the callback function that should handle the URL request. The line after that defines an optional validator callback. This is the callback to be used for validating application parameters.

Command Line scripts

If the application has functions that are called from the command line, then each function should have an entry in Callables.txt, that starts with command:. The entry should specify the name of the command that needs to be entered by the user from the command line. The next line defines the callback function for handling the command. For example if an application defines the command “Generate Site Map”, then it can be run with the command:

php index.php --application=[app-name] --command="Generate Site Map"

The Callables.txt file should contain the entry:

command: Generate Site Map
callback: {"object": "websitetools", "function": "GenerateSiteMap"}

This entry means that the function GenerateSiteMap of the object “websitetools” will be called, when the command “Generate Site Map” is entered. The config file: config\General.php, should contain the following line:

/** The custom commands */
$config['commands'] = array("Generate Site Map (generates site map of website)");

The above command defines the commands implemented by the application. These commands are shown to the user when he enters the command:

php index.php --application=[app-name] --command="Help"

This command lists all commands supported by the application.

The following code shows sample contents of the Callables.txt file:

url: ^/$
callback: {"object": "homepage", "function": "Generate"}

url: ^/articles(/view/\d+/[A-Za-z\-0-9\._%]+)?$
callback: {"object": "viewarticlepage", "function": "Generate"}

url: ^/articles/(tag|search)/[a-z\-0-9A-Z\._%]+?$
callback: {"object": "listarticlepage", "function": "Generate"}

url: ^/contact/form$
callback: {"object": "contactpage", "function": "Generate"}

url: ^/contact/add$
callback: {"object": "contactpage", "function": "SendContactMessage"}

command: Generate Site Map
callback: {"object": "websitetools", "function": "GenerateSiteMap"}

Template Engine

The Pak Php Framework includes a template engine which allows user applications to merge application data with HTML templates

Test Manager

The Pak Php Framework provides classes for testing code. Three types of tests are supported. White Box tests, Black Box tests and Ui (user interface) tests.

White Box tests are written like PhpUnit tests. The user defines test methods in a file. Each method name should start with “Test”. The test is then run from the command line. Assert statements may be used within the test methods to validate conditions.

Black Box tests are written by defining test data inside test files. Each method has its own test data file. The first line in the test data file gives the list of parameters separated by ’|‘. The last three entries in the first line gives the expected return value, the type of the return value and the rule used to validate the return value. The test data file can be auto generated for each method defined by the application. The command:

php index.php --application="[app-name]" --action="Generate Test Data"

will generate test data files for each user defined method. The test data files will be placed in the folder: test/testdata/{class-name}. The name of the file is same as the name of the method. Following is a sample structure of a test data file:

param_int_name1|param_int_name2|param_int_name3|return_name1|return_type|rule
1|1|7|{"count":7}|array|count

A UI (user interface) test allows the HTML of the response to be checked for errors. The validator.nu service is used to check if the HTML conforms with the HTML5 standard. Broken links are also checked as part of the Ui test. The test data for the Ui test is defined in a database table. See the structure of the table: pakphp_test_data. It has two main fields which are the URL and URL parameters. This data may be auto generated by setting the save_ui_test_data variable to true in config/Test.php. When this option is set to true, the framework saves the current URL and parameters to database

After a black box or white box test has been run, code coverage information for the test is displayed on the console and also saved to database. A summary of the test results is saved to database and file. A trace log of all function calls is also saved. The code coverage and function trace are generated using XDebug. The following screenshot shows the test results that are printed to the console after running black box tests:

The following screenshot shows the code coverage summary after running black box tests:

Utilities

The utilities folder contains classes that provide utility functions. These classes are used by the framework and may be used in user applications. See the Utilities Framework package for information on how to use the utility classes.

Vendors

The vendors folder contains third party libraries such as Twitter Bootstrap, JQuery, W3.CSS and QUnit. These frontend libraries can be used in html templates. Reusable widgets may be developed that allow developers to easily create attractive looking applications.

Installation

Run the command:

composer require nadirlc/pak-php-framework (Installation using Composer) OR
git clone https://github.com/pakjiddat/pak-php-framework.git (Download from [GitHub Repository](https://github.com/pakjiddat/pak-php-framework))

After the source code has been downloaded, create a MySQL database called pakjiddat_pakphp and import the contents of the file framework/data/pak-php-framework.sql to the database. This will create the database tables used by the Pak Php Framework. These tables are used for saving error data, access logs, test data and test results.

Examples

The helloworld sample application shows how to get started with the Pak Php Framework. To run the application, enter the name of the host used to access the application in the helloworld/Config.php file. Replace example.pakjiddat.pk with your-host-name. Also enter the database server credentials in helloworld/config/RequiredObjects.php. After that open the application in the browser.

For a more complex example you can download and install one of the following:

Developers Site. It is a simple website that allows web developers to publish their work
Islam Companion Website. It allows users to read the Holy Quran and Hadith. Also allows subscribing to Holy Quran and Hadith by email
Islam Companion Web API. It provides a RESETFul API for fetching Holy Quran and Hadith data

Utilities Framework

Sun, 10 Mar 2019 00:00:00 GMT

Introduction

The Utilities Framework is a set of PHP libraries that provide functions such as error handling, logging, emailing, fetching web pages, script profiling, database abstraction, encryption, template engine and more. It requires PHP 7.2 and above. The libraries are easy to use and can be used with custom applications and PHP Frameworks.

The Utilities Framework code is fully commented and compliant with the PSR-2 coding guidelines.

Features

The Utilities Framework has the following features:

Database Management

It provides functions for working with Databases. It consists of a Query Builder, Database Cache Manager and Database Log Manager. It also provides a Database MetaQuery Runner and Database Transaction Manager, based on PDO 2. #### Method Validation This allows validating method parameter values against information in the method’s Doc Block comments. This feature is provided by the Comment Manager package. 3. #### Error Management It allows application errors to be displayed using HTML template files. The template files can be easily customized. Default template files are provided for displaying formatted error messages for the browser and the command line 4. #### File and Folder Management It provides functions for fetching URL contents, checking if network connection works, searching for files with a folder and copying folder contents recursively 5. #### Email Handling It provides functions for sending email in plain text format and HTML format with file attachments. It is based on the Pear Mail library and hence supports sending email using SMTP server, PHP mail function and Sendmail library
6. #### Log Management It provides functions for saving and updating log data to database. It uses the PDO library for saving data and hence allows log data to be saved to all the databases that are supported by PDO 7. #### Template Engine It provides a template engine that allows separating the HTML layout code from the data. It also allows templates to be built recursively. This means that a template can consist of one or more templates, which can contain more templates. This allows complex website layouts to be divided in to simple layout files that are automatically combined by the template engine 8. #### Encryption It provides function for encryption and decryption data using the new LibSodium library, which is part of PHP >=7.2 9. #### String Utilities It provides functions for exporting data to RSS format, converting relative URLs to absolute, checking if string is valid JSON, HTML or Base64 encoded and more 10. #### Profiler It allows the memory usage and execution time to be measured between function calls 11. #### Cache Manager It allows data to be stored in a cache. It supports memory cache and database cache 12. #### Authentication It provides functions for authenticating users using HTTP digest authentication

The following screenshot shows the error message displayed by the error handler component:

The following screenshot shows the MySQL query log displayed by the error handler component as part of the error:

Installation

Run the command: composer require nadirlc/utilities-framework (installation using Composer) OR
Run the command: git clone https://github.com/pakjiddat/utilities-framework.git (Download from GitHub Repository)

Usage

All components of the Utilities Framework can be accessed using factory functions. To use a feature, we need to first create an object of the relevant component. For example: UtilitiesFramework::Factory(“email”, $parameters);. To send an email the following code can be used:

/* The Email class requires Mail and Mail_Mime pear package */
include_once ("Mail.php");
include_once ("Mail/mime.php");

/* Change the from and to emails to your email address */
$from_email       = "nadir@dev.pakjiddat.pk";
$to_email         = "nadir@dev.pakjiddat.pk";
/** The parameters for the email object */
$parameters       = array("params" => "", "backend" => "mail");
/* The Email class object is fetched */
$email            = UtilitiesFramework::Factory("email", $parameters);
/** The email is sent */
$is_sent          = $email->SendEmail($from_email, $to_email, "Utilitiesframework Test",
                        "<h3>test html content</h3>",
                        null,
                        array("file-path")
                    );
/** If the email was sent, then information message is shown */
if ($is_sent) echo "Email was successfully sent";
else echo "Email could not be sent";

Examples

The /examples folder contains example usage for each component.

Installing Editoria book publishing platform

Sat, 09 Mar 2019 00:00:00 GMT

Introduction

Editoria is a web based tool for publishing books. It allows editors and authors to collaborate on book publishing. Editoria is being developed by the University of California Press and the California Digital Library in partnership with the Coko Foundation.

Editoria’s goal is to provide a modern web based publishing platform that is more efficient, then current desktop based book publishing software. Editoria provides a web based workflow tool for styling, copyediting, author review, and proofreading.

Editoria is available as open source software and can be downloaded and installed by anyone free of cost. It is based on NodeJs library. In this article I will describe my experience with installing Editoria on Debian 9 (Stretch) operating system.

Installation

Editoria is based on several PubSweet components. It also uses PostgreSQL database. The Editoria installation guide describes how to install Editoria. The main installation steps are as follows:

Editoria requires NodeJs version 8.3 and above. Version 8.9 worked for me. Docker and Docker Compose are also required for running the PostgreSQL database server.

Installing project dependencies

The first step in installing Editoria is to clone the source code repository on GitLab using the command:

git clone https://gitlab.coko.foundation/editoria/editoria.git

The next step is to install the project dependencies using the Yarn package management tool for NodeJs. Yarn can be installed using the command: npm install -g yarn. To install the dependencies for the Editoria, run the command yarn from the root folder of the source code.

The next step is to change directory to application folder which is packages/editoria-app and create the configuration file: config/local-development.json. This was not possible since the packages/editoria-app folder did not exist. A similar issue was mentioned on the Editoria issue tracker. The solution as suggested in the issue was to first clone the repository https://gitlab.coko.foundation/editoria/ucp.git and then replace the text https://git@gitlab.coko.foundation/pubsweet/html-epub with https://git@gitlab.coko.foundation/pubsweet/html-epub.git, in the file yarn.lock.

Configuration

After that the dependencies should be installed with the command: yarn. The application folder is the root folder of the source code. Next the configuration file config/local-development.json needs to be created. This file contains credentials for configuring the INK server. The INK server provides an API for uploading Word files. Without the INK server the user will not be able to upload Word files. I did not install the INK server, so there was no need to edit the configuration file.

The next step is to create the configuration file, config/development.env. This file contains environment variables such as database server credentials, email server information and the INK server credentials.

The environment variables given in the file should be imported with the command: source config/development.env.

Database server configuration

Next the database server should be started with the command: yarn start:services. The command should be run from the root folder. It will start the Docker container containing the PostgreSQL server.

If the command yarn start:services fails with the error, that the root role does not exist, then the root role and an admin user should be created within the Docker container. The steps for creating the role and user are described in an answer to the question: PostgreSQL error: Fatal: role “username” does not exist on StackOverflow.

To create the database role and user, we first need to get the name of the container using the command: docker container ls. Next we need to enter the container command prompt with the command: docker exec -it {container name} /bin/bash.

Next we need to switch to user postgres and then select the template1 database from psql prompt. This can be done with the commands:

sudo su - postgres;
psql template1;

The template0 and template1 databases are default databases in PostgreSQL, that are used as templates for creating new databases. The default postgres database is intended to be used by users, utilities and third part applications. Next we need to create a role with super user privileges. This can be done with the command: CREATE ROLE root superuser;.

Next we need to create a user with the command: **CREATE USER username; ** and assign it root role with the command: GRANT ROOT TO username;. Next we need to give login access to the user with the command: ALTER ROLE root WITH LOGIN.

Once we have created the root role and user, we need to configure the database with the command: yarn resetdb. This will prompt for a user name and password. It will also create some database tables. The user name and password specified here should be noted and entered in the config/development.env configuration file. After the file has been edited, the environment variables in the file should be imported with the command: source config/development.env.

After that we should start the database server again with the command: yarn start:service. Next in a new terminal, we can start the main Editoria application with the command: yarn server. This will start the Editoria application on port 3000. We should then be able to access the Editoria from a browser.

Conclusion

Editoria is an innovative book publishing platform, that allows all people involved with book publishing to collaborate. It is a useful tool for speeding up the book publishing process.

Comment Manager

Sun, 03 Mar 2019 00:00:00 GMT

Introduction

The Comment Manager is a PHP script that allows parsing method Doc Block comments. It also allows validating parameter values and return values using information in the Doc Block comments.

Features

1. Method parameter validation

If you have added the following comment to a method:

@param string $my_param [val1,val2,val3]

Then the Comment Manager script can be used to check if the value given for the parameter, $my_param is in the list val1,val2,val3.

The script can validate range of numbers, parameter types, lists, emails, json and arrays. Method return values can also be validated based on information in DocBlock comments

2. Extracting method description

The Comment Manager script can be used to extract the description text given in method Doc Bloc comments. Long and short method description can be extracted

3. Validating method context

The Comment Manager script can be used to check if a class method is being called from the command line, from browser, as an API function or test function etc, based on information in the @internal tag

4. Custom Validation

The Comment Manager script allows validating parameter values using custom validation functions

5. Supported tags

Currently the Comment Manager script can be used to extract the following tags from DocBloc comments: @param, @version, @internal, @return and the long and short method description.

The Comment Manager script can be easily extended to extract other tags from method comments

6. Use in MVC frameworks

The Comment Manager package can be used in MVC (Model View Controller) frameworks for performing automatic validation of method parameters based on information in the method Doc Block comments

7. PSR Standard

The Comment Manager package code is compliant with the PSR-2 coding style

Example Use

Example 1

The following example shows how to call a function safely with parameter and return value validation:

/** The safe_function_caller closure is fetched from the CommentManager class */
$safe_function_caller       = CommentManager::GetClosure();

/** The parameters for the test function */
$parameters                 = array(
                                  "number1" => 30,
                                  "number2" => 10,
                                  "number3" => 10,
                                   "data" => array(
                                                 "type" => "integer",
                                                "random_string" => "The result of adding the three integers is: "
                                            )
                              );

/** The function that provides custom validation for the test function parameters */
$custom_validation_callback = array($this, "CustomValidation");

/** The callback */
$callback                   = array(
                                  "class_name" => get_class(),
                                  "class_object" => $this,
                                  "function_name" => "AddNumbers",
                                  "parameters" => $parameters,
                                  "context" => "cli"
                              );
try {                                          
   /** The test function is called through the safe function caller */
   $result                  = $safe_function_caller($callback, $custom_validation_callback);            
}
catch (\Error $e) {
    /** The error message is displayed and the script ends */
    die($e->getMessage());
}

/** The result of adding the numbers is displayed */
echo $result['random_string'] . $result['sum'];

In the above example, CustomValidation is a user defined function that performs custom validation. AddNumbers is the function to validate. It takes 3 numbers and a random string as parameters. It returns the random string as it is and also the sum of the three numbers.

The custom validation function validates the return value by checking if the length of the random string is within range. The type and value of the parameters is checked by the Comment Manager classes. See the example file CommentManagerTest.php for the complete example.

Example 2

The following example shows how to extract method DocBlock comments:

/** The Parser class is included */
require("Parser.php");
/** An object of the parser class is created */
$parser          = new Parser();
/** The method doc block comments are parsed */
$parsed_comments = $parser->ParseMethodDocBlockComments($class_name, $function_name);
print_R($parsed_comments);

In the above example, $class name and $function_name is the method to check. The following output is produced:

List of files

The Comment Manager package consists of the following files:

CommentManager.php. It provides a closure function for safe calling of class methods. It also provides functions for parsing and validating method parameters and return value
Parser.php. It provides a single function for parsing all method DocBlock comments
DescriptionParser.php. It provides functions for extracting long and short method description as well as the internal and version tags
ParameterParser.php. It provides functions for parsing the @param tag
Validator.php. It provides functions for validating method parameters and method return value using information in DocBloc comments
VariableValidator.php. It provides functions for validating different types of variables. For example integer, string, boolean and arrays

Installation

Run the command: composer require nadirlc/comment-manager (installation using Composer) OR
Run the command: git clone https://github.com/pakjiddat/comment-manager.git (Download from GitHub Repository)

Website performance testing

Tue, 19 Feb 2019 00:00:00 GMT

Introduction

The role of websites has evolved greatly over the last few years. Websites were initially a novel way of presenting information. Over the last few years websites have grown greatly in importance. Websites now may be regarded as the public image of all types of organizations.

A well developed informative company website is a great asset for the company. Several technologies related to websites have been developed over the years, such as website performance testing, search engine optimization, email marketing etc.

Given the importance of websites, it is important to ensure that a website loads correctly on all types of devices. This is no small task as it involves ensuring that all website components are being loaded correctly. For example CSS, JavaScript files should be loading quickly and in correct sequence. Images should be compressed so that they take less time to load. Web page HTML needs to conform to well known standards such as HTML 5.

The most commonly used tool for measuring website performance is the browsers developer tool bar. All major web browsers such as Google Chrome, FireFox and Internet Explorer have a built in developers tool box which provides detailed information on web page content.

Several online services such as GTmetrix and Google Pagespeed are available for measuring website performance. These services not only measure various performance metrics but also provide suggestions on how to improve the website performance. It is the job of the website developer to implement these suggestions and ensure good website performance. In this article I will focus on the suggestions for improving website performance.

Some of the recommendations given by these tools are:

Combine external JavaScript and CSS

This implies combining several JavaScript files into one file and several CSS files into one file. This reduces the number of requests sent by the browser and reduces the page loading time. The combined file may then be compressed or minimized which further reduces the loading time.

On WordPress based websites this recommendation can be implemented using a plugin such as JS & CSS Script Optimizer. This plugin supports combining and minifying JavaScript and CSS files.

Minimize redirects

This involves reducing the use of redirect URLs. Redirect URLs are those URLs that redirect to other URLs. This increases page load times.

Minimize DNS lookups

This implies reducing the number of host names that need to be looked up by the browser. This can be implemented by hosting all your web resources under a single host name. Use of a content delivery network (CDN) can also help in this regard.

Leverage browser caching

This is a common recommendation given by most of the tools. It implies making use of web browser caching. Web browsers can place resources such as CSS, JavaScript and image files in their local cache. By using resources in its local cache, the web browser has to make fewer requests to the web server.

To make use of browser caching you need to configure your web server so it instructs the web browser to cache files for certain duration. For the Apache web server this can be achieved by entering commands in htaccess file. WordPress provides caching plugins such as W3 Total Cache, which have several caching features, including browser caching.

Use a Content Delivery Network

This means using a Content Delivery Network for hosting static resources such as JavaScript, HTML, CSS and images. The benefit of this is that the content is delivered more reliable and quickly. CDN services such as Akamai, Cloudflare and Cloudinary can be used to host your static resources.

Cloudinary is a CDN service provider that has a very useful free plan. They also have a plugin for WordPress that allows content hosted on their servers to be accessed from WordPress websites.

Put scripts at the bottom.

This implies placing your JavaScript and CSS files at the bottom of your HTML page. It makes sense to delay loading CSS and JavaScript files that are not needed for rendering the page.

These files may even be loaded asynchronously. Loading asynchronously means the file is downloaded and processed while other operations are being carried out. This allows the web page to render more quickly. Delaying the loading of scripts may not always be desirable. For example CSS and JavaScript files needed for rendering a page should be placed inside the head tag so they can be downloaded and processed quickly.

Choose <link> over @import

This implies linking your CSS files using links tags instead off CSS @import statements. Using link tags is preferable because certain web browsers load CSS files that are linked using @import statement at the bottom of the page.

Gzip and compress components

This implies compressing web page content before it is sent by the browser. Content that is compressed takes less time to be transmitted and also reduces the network bandwidth usage. The W3 Total Cache WordPress plugin has support for compressing content. In Apache web server it can be achieved using the mod_gzip module. Once this module has been activated it can be configured using htaccess file.

Optimize and correctly display images

This implies ensuring that your images are compressed and are not larger than the required size. Images tend to be large in size and are one of the main bottlenecks of website performance. It is important to compress an image and resize it before serving it on your website.

In WordPress this can be achieved using plugins such as EWWW Image Optimizer or Smush.it. Both these are excellent plugins that can resize and compress the files in WordPress media library. Alternately you can use a free online image compression tool.

Minify HTML, CSS, and Javascript

This implies compressing HTML, CSS and JavaScript files before serving them. This makes the file size small and reduces the time needed to download the file. The W3 Total Cache plugin supports file compression.

Use CSS Sprites

A CSS Sprite is one large image that contains smaller images. CSS code is used to display images within a CSS sprite. CSS Sprites are specially useful for combining background images into a single image. The benefit of this is that it reduces the number of requests for individual images.

Monitor plugin performance

Remove the plugins you don’t need. Having too many plugins on your site greatly reduces performance. You can also use the P3 performance profiler plugin to check which plugins are slowing down your website. This tip was suggested in a useful blog post titled ”Speeding up WordPress load from 4.23s to 1.33s [Case Study]“.

Use online validation tools

The World Wide Web Consortium (W3C) is an organization that manages HTML standards. It has an excellent on-line tool that can be used to validate HTML documents.

In WordPress you can validate each page and post of your website using the validated plugin. WordPress also has a useful plugin called Broken Link Checker, that can be used to check your website for broken page links.

Conclusion

Ensuring that the HTML of your website complies with web standards is an important consideration for any website. A website that is standards compliant will not only receive a higher ranking in Google search results but will also render more reliably across different web browsers.

Some useful links

Performance Unleashed: How To Speed Up WordPress Load Times is a useful article that describes how to improve WordPress load times. The optimization tips given in the article can also be applied to websites that are not based on WordPress.
Best WordPress Hosting: Top 10 Comparison & Reviews is another useful article which reviews the best WordPress Hosting services.
17+ Best Web Hosting Services in 2020 (Providers Ranked by Industry, Use Case, and Price) is an excellent article that provides information about the different web hosting services. The article provides information about the best web hosting providers, the fastest web hosting plans, best WordPress hosting services for Bloggers, the best web hosting for small businesses and more.

Tracking Issues with Mantis Bug Tracker

Wed, 02 Jan 2019 00:00:00 GMT

Introduction

Mantis Bug Tracker is web based software that allows keeping track of issues. It is used mainly for tracking bugs in software, but may be used as a general purpose issue tracker for all types of projects.

Mantis Bug Tracker was first released in 2000. It is supported by an active software community and is stable and user friendly.

It is written in Php language and supports several databases including MySQL, MSSQL and Oracle. In this article, I will describe my experience with installing and using Mantis Bug Tracker for tracking software related issues. I will also describe my experience with upgrading Mantis and migrating issues from Jira to Mantis

Installation

Installation of Mantis Bug Tracker is very simple and takes a couple of minutes. It is well documented in the online documentation for Mantis. The pre-requisites for using Mantis Bug Tracker are a Web Server, Php and a database server such as MySQL.

To install Mantis Bug Tracker, we need to first download the files from SourceForge, and then extract them to the document root folder. Next we need to run the installer from the web browser by visiting the url: http://your-domain/path-to-mantis/.

The web based installer will ask a few questions such as the name of the database server, user name and password for the database user and the database table name prefix. After entering the required information, we need to press submit and the installer will create the database tables. This completes the installation process.

The process for upgrading Mantis Bug Tracker is identical to the installation process. To upgrade an existing installation of Mantis, we need to simply install the new version of Mantis and specify the information for the existing Mantis database during the installation.

Once the installation has completed, we can login with user: administrator and password: root. We will be prompted to remove the admin installation folder and change the default password. For those interested in trying out Mantis, a demo of the current version is provided.

Configuration

The Mantis Admin Guide provides useful information about Mantis. To get started with Mantis, we need to first create a project. Each project contains issues.

Issues

The issue is the main feature of Mantis. It contains fields such as description, summary, reporter, assigned to, status, resolution, steps to reproduce the issue, additional issue information and more. We can also add unlimited notes, attachments, tags and custom fields to issues.

Projects

A project is a container for issues. Each project can have its own sub projects, custom fields, categories and versions. The list of users with access to the project can also be given.

Change Log

A useful feature of Mantis is the Change Log. It allows us to keep track of our projects. The Change Log shows the list of issues grouped by project version. To use the Change Log we need to first add versions to our project. When reporting an issue, we can select the project version under the “Fixed in Version” and “Product Version” fields. The issue will then appear under this project version on the “Change Log” page.

Road Maps

The Road Map shows issues for future versions of our projects. Each project version may be marked as released. A project which is not marked as released will show up on the Road Map page along with the relevant issues.

The Road Map page shows a progress bar describing how much progress has been made in completing the project. The progress is calculated by the number of issues completed in the project version. To configure an issue so it shows on the Road Map, we need to select the project version under the “Target Version” field. This field shows only project versions that have not been released.

Email Notifications

Another useful feature of Mantis is the ability to get email notifications on issue changes. These notification emails may be configured at a detailed level for user roles. For example each time an issue is updated, an email may be sent to the project developers describing the change.

The Mantis configuration file allows detailed configuration of Mantis including enable/disable email notification, email server settings for SMTP email, Logo and title, fields to show/hide on the add, edit and view issue pages and much more. A useful feature of Mantis is that it allows configuration of almost all features at a detailed level.

Plugins

A useful feature of Mantis is the ability to customize its features using plugins. The list of plugins for Mantis is listed on the Mantis wiki. For most plugins, the installation is as simple as downloading and extracting the plugin files to the plugins folder. After that the plugin needs to be enabled from the plugins page.

Migrating issues from Jira to Mantis

Jira is a feature rich Issue Tracker from Atlassian. Like Mantis it has been in use for several years and is in active development.

In the article, Installing Atlassian products on Google Cloud, I described my experience with using Jira. Jira has more features than Mantis, but its main drawback is that it takes a lot of server resources. In order to reduce our billings on Google Cloud, we decided to migrate from Jira to Mantis.

To migrate issues from Jira to Mantis, I first selected a project in Jira and enabled the columns that I wanted to export. Next I clicked on the option for exporting to CSV format with current fields. This exported all the issues in the project to a CSV file. I did this for all the projects that I wanted to export.

Next, I installed the Mantis CSV Import plugin, which allows importing issues from a CSV file. To import the Jira issues, I simply uploaded the CSV files one by one.

After each upload, I was asked to select the field mappings for the issues. The field mappings option allows selecting the fields on the issue page that corresponds with fields in the CSV file. After selecting the field mappings, the issues were imported. The imported process completed but gave an “Internal Application” error, which I posted to the issue tracker for the plugin.

Conclusion

Mantis Bug Tracker is very user friendly and easy to use. It is stable, well documented and is supported by an active community. Its useful for keeping track of all types of issues such as those related to Server Management, Website Maintenance and Website Development.

JavaScript arrow functions and the let keyword

Sun, 25 Nov 2018 00:00:00 GMT

JavaScript arrow functions are unnamed functions which are best used as non-method functions. They provide a short hand notation for using functions. An arrow function consists of a list of function parameters followed by an arrow which is followed by an expression to be returned or a code block. The following code shows how to create an arrow function

let ToUpperWrapper = (value) => {value.toUpperCase();}

ToUpperWrapper("test");

The let keyword is used to define a variable with a local scope. A variable declared using let can only be used in the block, expression or statement in which it is used as well as in sub blocks. The var keyword allows a variable to be used globally or within a function block. Variables created with var can be accessed any where within the enclosing function

See the article Arrow functions for more information about using JavaScript arrow functions

Using JavaScript modules

Sun, 25 Nov 2018 00:00:00 GMT

JavaScript Modules are an implementation of ES6 Modules. They allow a useful way to organize and reuse JavaScript code. JavaScript Modules consist of a set of features that provide module functionality.

The export keyword allows exporting functions, constants, variables, class etc. For example:

// lib.mjs
export function Test(param1) {
    return "Hello World !";
}
export const testdata = "Test"

The import keyword can be used to import exported functions and variables. For example:

// main.mjs
import Test from ./lib.mjs;
Test("test argument");

In the above example, the lib.mjs must have a relative path or it should be specified with http(s).

Module files are loaded like normal JavaScript files but with the type attribute set to “module”. The following example shows how to load a module:

<script type="module" src="main.mjs"></script>
<script nomodule src="fallback.js"></script>

In the above example, the fallback.js file is loaded only by browsers that do not understand the JavaScript module standard. Browsers that understand module standard ignore the fallback.js script.

Modules are different from normal scripts in that they are always executed in strict mode. HTML style comments are not supported in modules. Instead single line comments should be used. Module scripts are loaded with Cross Origin Resource Sharing (CORS). This means the HTTP response that delivers the module code should include the HTTP Header: **Access-Control-Allow-Origin: *** or similar.

Another feature of JavaScript modules is that they are loaded by the browser in deferred mode by default, which means they are downloaded while the page is being parsed. JavaScript module files should have the .mjs file extension, but it is not required. The important thing is that the file is served with content-type of text/javascript;.

Dynamic import allows loading JavaScript modules from within scripts. The following example shows how to load JavaScript modules dynamically:

<script type="module">
  (async () => {
    const moduleSpecifier = './lib.mjs';
    const {repeat, shout} = await import(moduleSpecifier);
    repeat('hello');
    shout('Dynamic import in action');
  })();
</script>

See the article Using JavaScript modules on the web for more information about using JavaScript modules.

Freelancing over the Internet - Drawbacks and Benefits

Tue, 13 Nov 2018 00:00:00 GMT

Introduction

In this article I describe my experiences and conclusions of working as a Freelancer over the internet. I will describe the benefits and drawbacks of Freelancing in general with an emphasis of Freelance I.T work.

Freelancing over the Internet is a career option pursed by many people these days. Its popular in both developing and developed countries. The advancement of Internet and Communication related technologies has led to an increase in the number of Freelance I.T workers. It has also created new challenges for individuals and society.

Specialized software tools such as online portals, social networking platforms, voice and video conferencing, support forums etc have led to the creation of virtual communities, where people with common interests meet and share ideas.

Freelancing over the Internet is made possible by specialized software that is used by the community members to interact with one another. For example Freelancers use communication tools such as Skype or an Instant Messaging client for communication with each other. Inspite of the rapid development in online communication tools, email remains a popular choice for communication between Freelancers.

Online payment services such as PayPal, Skrill and Payoneer allow Freelancers to receive payment from their contractees. Online project management tools such as Redmine and Open Project are used to record the details of all work done by the Freelancers.

Online Job Portals such as Upwork, Elance and Toptal allow Freelancers to choose the type of work that best suites them

Benefits of Freelancing over the Internet

Freelancers in most cases have the freedom to choose their work hours and work place. This means they have more flexibility in planning their day to day activities. Many offices have a strict working culture and do not encourage religious and charitable activities. A Freelancer who works from home can divide his time between family, work, religious and charitable activities. He can choose when to exercise, when to sleep, when to eat, when to work etc.

Another benefit of working as a Freelancer is that one does not have to commute to work and therefore does not have to spend as much on transportation and clothes. Also Freelancers suffer less from work related stress.

Working from home is the new normal and is part of the modern gig economy. The article 6 signs that the future of work is here today describes how elastic staffing and working from home are affecting the modern corporate work environment.

Drawbacks of Freelancing over the internet

Freelance workers who work remotely cannot interact with their contractees in person. This is a major drawback since it means they cannot get as good an understanding of their work as compared to a person who works onsite.

Working onsite also means you get to learn from the experiences of others. Interacting with others professionally helps in developing personality and social skills. It also means the worker has a much better idea of what his employer is asking of him. He is thus able to contribute more effectively towards the achievement of his organizations goals.

Freelancers have the freedom to choose who they work for. They can work for anyone in any part of the world!. This has implications for society as well as the individual. If someone works for an employer who lives in another country and he spends over 40 hours per day working for him, then it would seem that part of him is a member of his physical society and part of him is a member of a virtual community.

I think this is not a fair assessment. An individual is either part of a physical society or he is not. He must not think of imaginary virtual worlds. He should regard his virtual community as another set of obligations not different from those of his physical world. He must focus on meeting those obligations.

Opportunities for Freelance I.T workers

Being a Freelancer not only means you can choose when and where to work, it also means you can choose who to work for. A Freelancer may work part time for a contractee and spend his remaining time contributing to worthy causes. He may also run his own business along side his Freelance work.

There are numerous opportunities available online for Freelance workers. For example a Freelancer who needs money, can simply work for more people. Of course he must be careful about fulfilling his contractual obligations. Before working for a contractee online, he must ensure that he understands the terms of the contract between him and his contractee. If an understanding cannot be developed, then the contractor must stop working for that contractee and look for other opportunities

Starting you own online business

Starting your own business is an attractive concept. Running a physical business is beyond the capability of most people. Not everyone is blessed with the mental and physical capabilities that are required to run a business.

The internet has allowed people to overcome this barrier. Anyone who can read and type, can build a respectable online presence. However this does not mean that everyone can succeed over the internet. Succeeding in any occupation requires hard work, dedication and patience. The internet I believe only provides new opportunities to excel. The hard work of course still has to be done.

Starting an online business requires very little capital and initial effort. All you need is a website that displays your products and services. These products and services dont even have to exist in the real world!. To attain true success however, the online business must be developed incrementally and improved over time.

For example, a Freelancer may start an online blog and transform it into a respectable online business. He may start with publishing blog posts describing his Freelance work. He may also publish the source code that he has written on his own website.

He may develop his own software components and make them available for others. While this may not make him a millionaire over night, it will bring him valuable feedback from which he can learn and improve. Fruitful and productive interaction over the internet, I believe is important for a person to understand how to benefit from the internet.

Drawing more visitors to your blog is a good idea, since it brings feedback which can be used to improve and build on your ideas of running an online business. How to start a business blog is a useful info-graphic that pictorially describes how drawing visitors to your website can help you grow your online business.

Cloud related technology has had a major impact on businesses. The Cloud offers various tools for collaboration, communication, data storage and computing that can benefit all types of businesses. The article 10 Tips How The Cloud Can Make Your Company More Productive in 2015 is a useful article that describes how Cloud related technologies can benefit businesses.

While the Cloud has created new opportunities for businesses, its usage has created new challenges for society. Excessive usage of any technology is not a good thing and the Cloud is certainly no exception. Excessive usage of Cloud related technologies has created health and social problems, which must be identified and corrected.

Conclusion

Freelancing over the internet is a popular occupation that has grown in leaps and bounds in recent years. The advancement of technology has led to specialization in the work that is done remotely. For example website development and maintenance, software development, help desk management and call center management are some of the sub areas of Freelance work.

The online job Marketplace freelancer.com lists over 900 job categories. Before choosing a career in Freelancing, I believe it is important to be aware of the pros and cons of Freelancing as well as the pros and cons of the technology behind it.

ES6 brings new features to JavaScript

Thu, 08 Nov 2018 00:00:00 GMT

The ES6 standard has introduced several changes to JavaScript. These changes include Classes, Proxies, Promises, Modules, Iterators and Generators and more

The for of syntax allows iterating over an array. The following example shows how it works:

var array = [4, 5, 3, 7, 9];

//'i' refers to the values of the array indexes
for(var i of array)
{
    console.log(i); //4, 5, 3, 7, 9
}

The spread operator which is ”…” allows expanding an array. It can be used to allow functions to take indefinite number of arguments. For example:

//args variable is an array holding the passed function arguments
function function_one(...args)
{   
    console.log(args);
    console.log(args.length);
}

function_one(1, 4);
function_one(1, 4, 7);
function_one(1, 4, 7, 0);

ES6 also allows functions to return multiple values. It allows a function to return a numerically indexed array. Its values can be assigned to multiple variables. For example:

function function_name()
{
    return [6, 2, 3, 4, 5, 3]; //here we are storing variables in an array and returning the array
}

var a, b, c, d, e, f;

[a, b, c, d, e, f] = function_name();

The const keyword is used to declare constant variables. A variable prefixed with const, cannot be updated. Also we cannot declare more than one variable with same name and scope as constant

See the article ECMAScript 6 Complete Tutorial for more information about the new ES6 features

Using await and async with Promises

Thu, 08 Nov 2018 00:00:00 GMT

The async keyword is used when declaring functions. A function declared with async must return a promise. If a function is declared with async and it does not return a promise, then the JavaScript engine will automatically convert the return value of the function to a promise object. For example the following functions are both valid async functions:

async function f() {
  return Promise.resolve(1);
}

async function f() {
  return 1;
}

The await keyword is used to wait for a promise to resolve. It causes code execution to wait until the promise has been resolved. The await keyword can only be used inside functions declared with async. It cannot be used inside regular functions. The following example shows how to use the await keyword:

async function Test() {

  let promise = new Promise((resolve, reject) => {
    setTimeout(() => resolve("done!"), 1000)
  });

  let result = await promise; // wait till the promise resolves

  alert(result); // "done!"
}

Test();

See the article Async/await for more information about using await and async with JavaScript Promises.

Using generators, classes and objects in JavaScript

Thu, 08 Nov 2018 00:00:00 GMT

JavaScript Objects

All values in JavaScript are objects except for primitive values which have no methods and properties.

JavaScript defines 5 primitive data types, which are: string, boolean, null, undefined and number.

A JavaScript object is a collection of named values which are called properties. An object property can be a primitive, function or an object.

An object method is an object property containing a function definition.

Examples of creating JavaScript objects:

var person = {firstName:"John", lastName:"Doe", age:50, eyeColor:"blue"};

var person = new Object();

/** The Object.create creates an object from an existing object */
const me = Object.create(person);

JavaScript objects are said to be mutable or changeable. When an object is assigned to another object, the assignment is done by reference. A new object is not created.

All JavaScript objects are descendants of the built in Object. All objects inherit properties that are in Object.prototype. These properties may be overridden by the objects. For example toString() method.

JavaScript classes

ES6 introduced a new keyword class, which allows creating classes using syntax that is common to most programming languages. The class syntax also allows inheritance. The class syntax is just a wrapper for object.prototype. It is implemented using the built in object.prototype property.

A major different between function declarations and class declarations is that functions are hoisted while classes are not. When we say that function declarations are hoisted, it means that a function can be called before it is defined, while a class cannot be used before it is defined. Following example shows how to declare and use a JavaScript class:

class Rectangle {
  constructor(height, width) {
    this.height = height;
    this.width = width;
    this.area = (width *height);
  }
  get area() {
    return this.calcArea();
  }
  set area(new_area) {
    this.area=new_area;
  }
}
const p = new Rectangle(10, 4);
let area = p.area;
p.area   = 20;

The constructor keyword defines a function that is automatically called when an object is instantiated. The get and set keywords allows functions to be called when a property is read or updated.

Functions prefixed with static keyword are called on the class. They cannot be called on an instance of the class. The extends keyword allows a class to inherit properties and methods from another class. The super function should be the first line of code in the class constructor. It calls the parent class constructor.

The W3school website provides useful information about JavaScript classes and objects.

Using Generators in JavaScript

Generators were introduced in ES6. One of the benefits of Generators is that they allow efficient handling of large arrays. JavaScript implements generators using the yield keyword and function* function declaration.

The function* denotes a generator function. Within the function, the yield keyword can be used. The following code describes how to use generators in JavaScript:

function* collection_name()
{
    yield 1;
    yield 3;
    yield 5;
    yield 7;
}

for(let count of collection_name())
{
    console.log(count);
}

See the part 10 of the tutorial ECMAScript 6 Complete Tutorial for more information about using Generators in JavaScript

Using JavaScript promises

Mon, 05 Nov 2018 00:00:00 GMT

A Promise is defined as an object representing the eventual completion or failure of an asynchronous operation. When creating a Promise object, we have to pass a function called executor as the constructor parameter. The function should accept two functions as arguments called resolve and reject.

The resolve function should be called by the executor to resolve the Promise. The reject function should be called to reject the promise. The following code shows how to create a Promise object:

var promise1 = new Promise(function(resolve, reject) {
  setTimeout(function() {
    resolve('foo');
  }, 300);
});

promise1.then(function(value) {
  console.log(value);
  // expected output: "foo"
});

The above code starts a timer for 300 ms. When the timer expires, the Promise is marked as resolved. Once the Promise has been resolved, a function is called which prints the value passed to the resolve function.

The then function is used to call functions that are called after a Promise has been resolved or rejected. It takes two parameters, the function to be called after the Promise has been resolved and the function to be called after the Promise has been rejected. It returns a Promise object. This allows chaining functions that need to be called asynchronously.

The catch function allows exceptions thrown inside the executor function to be handled. It takes a single parameter, which is the function to handle the error. The following example shows how the catch function works:

var promise1 = new Promise(function(resolve, reject) {
  throw 'Uh-oh!';
});

promise1.catch(function(error) {
  console.log(error);
});

See the article Promise for more information about using JavaScript promises.

Using Web Components in JavaScript

Mon, 05 Nov 2018 00:00:00 GMT

Web Components is a suit of different technologies that allow creating and using reusable custom HTML elements. Currently it is based on the three main features which are: Custom Elements, Shadow DOM and HTML Templates.

A Web Component is created by registering a class which defines the component. The component can then be used in two ways. It can either extend the functionality of an existing element such as a paragraph tag, or it can define a custom element with its own name.

A custom element can be defined using the following:

customElements.define('word-count', WordCount, { extends: 'p' });

The first argument is the name of the component. The second argument is the name of the class which defines the Web Component. It should be a valid ES6 class name. The class may extend HTMLElement or another HTML class such as HTMLParagraphElement. The third argument is an optional argument which specifies the HTML element that the custom element is based on.

The Shadow DOM API allows the custom element to keep its markup, style and code separate without effecting other parts of the page. The attachShadow allows a shadow root to be attached to any DOM element. Once the shadow root has been attached, it can be used like any other DOM element. For example, elements and styles can be added to it using JavaScript. The following example shows how to create the shadow root and attach it to the custom element:

// Create a shadow root
var shadow = this.attachShadow({mode: 'open'});

The mode parameter indicates whether the Shadow DOM should be accessible to the main page. For example the video tag contains a Shadow DOM which is in closed mode. Which means we cannot access its shadowRoot property.

Templates and Slots are another part of Web Components. A template is an inert block of code which is not directly rendered by the browser. It is defined using Template tag. The following code defines a template:

<template id="my-paragraph">
  <p>My paragraph</p>
</template>

A template is rendered by fetching its contents and appending it to a DOM element using JavaScript. The following shows how to use a template:

let template = document.getElementById('my-paragraph');
let templateContent = template.content;
document.body.appendChild(templateContent);

Slots provide placeholders for template values. Slots are defined using the slot tag. A slot tag has a name attribute which is used to access the slot from the custom element tag.

Slots are first defined inside the template tag using the slot tag. Then they are used inside the custom element tag by using the attribute slot on an HTML element such as span or div. The value of the slot attribute is the name of the slot tag defined in the template.

See the article Web Components for more information about how to use Web Components in JavaScript.

Generating free SSL certificate from LetsEncrypt on localhost

Wed, 31 Oct 2018 00:00:00 GMT

If your website hosting allows uploading SSL certificates but does not allow SSH access and you need a SSL certificate for your website, then a free SSL certificate from LetsEncrypt is a good option.

First we need to install the SSL certificate on a remote host, for example localhost. To install the SSL certificate we need to first install the Certbot Acme client. The Certbot installation guide for Debian (Stretch) on the Certbot website describes how to install Certbot on Debian Stretch. Similar guides are available for other Unix based operating systems. For example the following command will install Certbot on Debian (Stretch): sudo apt-get install python-certbot-apache -t stretch-backports.

Next, we need to obtain the SSL certificate. This can be done by issuing the command: sudo certbot certonly —manual. We will then be asked to enter the name of the domain. After that the screen will show some text and the name of a file. We will have to add the text to the file and then upload it to the server at the specified path. Next the LetsEncrypt server will verify the existence of the uploaded file and will issue the SSL certificates to our Certbot client. Usually the SSL certificates are saved to the folder: /etc/letsencrypt/live/. Next we need to upload the private key and SSL certificate to the website control panel. After this our website should be accessible over SSL.

See the Certbot documentation for more information.

How To Change Account Passwords on an OpenLDAP Server

Fri, 19 Oct 2018 00:00:00 GMT

To change the LDAP admin password, we need to first save the rootdn account information. We can do this with the command:

ldapsearch -H ldapi:// -LLL -Q -Y EXTERNAL -b "cn=config" "(olcRootDN=*)" dn olcRootDN olcRootPW | tee }/newpasswd.ldif

Next we need to generate a hash of the password using the slappasswd utility. The password should be appended to the LDAP configuration file. We can do this with the command:

/usr/sbin/slappasswd -h {SSHA} >> }/newpasswd.ldif

Next we need to place the generated hash password in the current location in the ldap configuration file. The new file should look similar to the following:

dn: olcDatabase={1}hdb,cn=config
#olcRootDN: cn=admin,dc=example,dc=com
changetype: modify
replace: olcRootPW
olcRootPW: {SSHA}lieJW/YlN5ps6Gn533tJuyY6iRtgSTQw

The hdb string is the type of LDAP database. It can have other values such as mdb.

After that we need to apply the changes using the command:

ldapmodify -H ldapi:// -Y EXTERNAL -f }/newpasswd.ldif

This changes the password in the administrative section of the LDAP configuration

We also need to change the password for the user. To do this we need to update the configuration file so it looks like following:

dn: cn=admin,dc=example,dc=com
changetype: modify
replace: userPassword
userPassword: {SSHA}lieJW/YlN5ps6Gn533tJuyY6iRtgSTQw

After that we need to apply the changes using the command:

ldapmodify -H ldap:// -x -D "cn=admin,dc=example,dc=com" -W -f }/newpasswd.ldif

The article How To Change Account Passwords on an OpenLDAP Server describes how to change account passwords on an OpenLDAP server.

INADEQUATE_SECURITY - SSL Cipher problems with HTTP2

Fri, 19 Oct 2018 00:00:00 GMT

HTTP2 requires encryption over SSL. The recommended SSL protocol for HTTP2 is TLS 1.2. Ciphers are encryption algorithms used by SSL protocols.

Certain devices specially old ones such as old Android mobile phones or old operating systems like Windows XP, support outdated SSL Ciphers that are not supported by HTTP2 protocol.

The TLS 1.2 Cipher Suite Black List shows the list of all SSL Ciphers that are not supported by the HTTP2 protocol. If a web browser running on an old operating system requests a SSL Cipher that is on the list of unsupported Ciphers, then the HTTP2 implementation of the browser should reject the connection to the server with INADEQUATE_SECURITY error message

To test if the connection between your device and a web server is over a reliable and secure HTTP2 connection, go to https://www.ssllabs.com/index.html and enter the url of the website

See the discussion nginx HTTP/2 support cipher problems on the official Plesk forum for more information

Securing Plesk using the Plesk Advisor extension

Fri, 19 Oct 2018 00:00:00 GMT

Introduction

Plesk is a useful control panel for Windows and Linux based servers. It is used for managing web hosts, users, server security, email and more. Plesk is a commercial application with different pricing plans starting at 9.16 USD per month. Users can try Plesk for free for up to 2 weeks. The Plesk website describes the different Plesk features.

Recently I had the opportunity of securing a Plesk based server. In this article I will describe my experience with configuring the following Plesk security related features: “Plesk Watchdog”, “Plesk Advisor”, “Plesk Server Health Monitor”, “HTTP2”, “Fail2Ban” and “ModSecurity”.

Securing Plesk

A server running Plesk should be configured from within Plesk. Installing custom scripts and applications outside Plesk is possible, but not recommended as it can conflict with scripts used by Plesk. Plesk has a number of built in features that can be used to secure the server. It also provides several extensions or addons for configuring server security. Following are some useful features/addons that can be configured:

Plesk Watchdog

The Plesk Watchdog is an extension for Plesk.

It can be installed by clicking on Extensions on the left menu and then selecting Watch Dog. The Plesk Watchdog allows scanning the server for malware, viruses and rootkits. It allows monitoring hard disk usage as well as different services managed by Plesk.

The Plesk Watchdog uses the Monit tool for monitoring services. It scans for viruses and malware using RKHunter tool. The Plesk Watchdog allows monitoring services such as Nginx, Apache web server, Postfix SMTP server, MySQL server and all other extensions that can be managed from Plesk.

Each service is monitored at an interval of 5 minutes by default. If a service is found to be stopped, then the Plesk Watchdog will start the service and will send an email to the server administrator. The user can configure the monitoring interval as well as disable service monitoring.

The commands for starting and stopping each service can also be configured. This option was very useful, because we had installed the latest version of PHP FPM and the Plesk Watchdog was using incorrect commands for monitoring the service. Fortunately it was easy to change the monitoring commands.

The Plesk Watchdog also allows monitoring the free disk space. It allows running a custom command if the free disk usage or number of files exceeds a certain threshold value.

The Plesk Watchdog also allows scanning the server for malware, root kits, Trojans and other malicious software. It uses the Rkhunter tool for this purpose. The security scan can be configured to run daily, weekly, monthly or disabled. After the scan completes a summary report is sent to the configured email address. A summary report is also sent containing overview of the security scan and service status. This report can also be sent daily, weekly, monthly or disabled. The Plesk Watchdog also shows statistics containing CPU and RAM usage by different services.

Plesk Advisor

The Plesk Advisor is a useful tool that suggests options for improving server security and performance. The suggestions are basically Plesk extensions or features that can be used to improve the server security and performance.

The Plesk Advisor also provides an overall score for the server which indicates how well the server security and performance has been configured. The higher the score, the more secure the server.

The suggestions given by the Plesk Advisor can be enabled or disabled with a single click. Some of the recommendations by the Plesk Advisor include:

Secure Websites With SSL/TLS Certificates. It allows enabling/disabling SSL certificates for websites.
Configure Scheduled Backups allows configuring backup of databases, domains and configuration.
Configure ModSecurity & Fail2ban allows enabling/disabling the built in Plesk features Mod Security and Fail2Ban.
Configure the Plesk Firewall allows controlling which users and IP addresses have access to Plesk.
Disallow Weak Passwords allows disabling the use of weak passwords.
Enable HTTP/2 allows enabling/disabling HTTP2
Switch On Automatic Updates allows the operating system updates to be installed automatically.

Plesk Server Health Monitor

The Plesk Server Health Monitor allows us to monitor CPU, Hard Disk and Memory usage. It is available as a Plesk component and can be installed by going to “Tools and Settings” and then “Updates”.

The Plesk Server Health Monitor allows us to monitor the CPU usage by each Plesk service. It provides current values as well as trend values in the form of graphs. For example it provides graphs that show how much CPU Apache web server has used. The CPU usage can be viewed for the last 3 hours, 1 day, 3 days and 1 week. Some of the monitored services are MySQL, PostFix and Nginx.

The Plesk Server Health Monitor also allows us to monitor disk usage and current memory usage. It provides details on overall CPU usage such as average load, number of processes, zombie processes etc. The Plesk Server Health Monitor also allows us to monitor the current number of network packets sent and received per second. A detailed graph shows the trend over different time intervals.

Emails notifications and alarms can be configured for the parameters being monitored. To configure emails notifications and alarms, we need to first click on “Download Configuration File”. This will download the configuration file for the Plesk Server Health Monitor.

The configuration file is a XML file that allows setting threshold values and email notifications for each parameter being monitored, such as CPU usage by Apache web server.

There are two types of alarms that can be configured. A yellow alarm indicating that resource usage is high and a red alarm indicating that the resource usage has reached a limit.

The Plesk Server Health Monitor does not automatically detect changes in server hardware. The changes are detected only once when the component is first installed. In case of hardware changes such as increase in RAM, the Plesk Server Health Monitor configuration needs to be updated by clicking on “Detect Hardware Changes”.

HTTP2 configuration

HTTP2 is a major version of the HTTP network protocol. It followed HTTP/1.1 protocol. It is based on the experimental SPDY protocol originally developed by Google. As of October 2018, 30.5% of the top 10 million websites supported HTTP/2. HTTP/2 is supported by most major browsers.

Its main features are improved performance and security. Performance improvement is achieved by several improvements such as header compression, multiplexing several requests and responses over a single TCP connection and server push.

HTTP/2 provides improved security by requiring the TCP connection to be encrypted. Although HTTP2 protocol supports unencrypted connections, all major browsers only support HTTP2 over SSL/TLS connection. TLS 1.2 is the recommended SSL protocol for HTTP2 connections.

HTTP/2 Server Push allows the server to send resources to the client without waiting for the client to request the resources.

For example if a browser sends a request for a web page index.html containing style.css and script.js, the server which implements HTTP/2 Server Push will send the style.css and script.js files to the browser along with the contents of index.html. This greatly reduces the web page loading time. HTTP/2 Server Push is supported by Nginx version 1.13.9 onwards and Apache version 2.4.17 onwards. HTTP/2 support for Apache is provided by the mod_http2 module.

Plesk support for HTTP2 protocol requires the latest version of Nginx web server.

Once Nginx has been installed, HTTP2 support can be enabled with the command: plesk bin http2_pref enable. To test if HTTP2 has been enabled for your website, go to the website: https://tools.keycdn.com/http2-test and enter the URL of your website. Also the Chrome Developers Toolbox has a column called protocol that indicates the HTTP protocol used to load each resource.

Fail2Ban

Fail2Ban is a command line tool for banning suspicious IP addresses. It scans log files such as web server and ssh server log files and detects abnormal user activity such as too many failed login attempts or several web page error messages. It then updates the system firewall such as Iptables and prevents further access from the IP address for a certain time period.

Plesk provides protection against brute force attacks using the Fail2Ban component.

The component first has to be installed. Next it has to be enabled, by going to Tools & Settings, then IP Address Banning (Fail2Ban) and then Enable intrusion detection. Next select the Enable intrusion detection checkbox. This will activate the Fail2Ban service.

The settings page for Fail2Ban allows us to set the IP address ban period, Time interval for detection of subsequent attacks and the Number of failures before the IP address is banned. The IP address ban period, is the time after which a banned IP address is automatically unbanned.

The Time interval for detection of subsequent attacks is the time interval during which the number of unwanted actions from an IP address are counted. The Number of failures before the IP address is banned is the number of unwanted actions after which the IP address is banned

The Plesk Fail2Ban component is based on the concept on Jails. A Jail is a set of rules covering an individual scenario for example SSH. A Jail is a combination of a filter and one or more actions. A filter is a regular expression that is applied to log files in order to determine suspicious activity.

Actions are commands that are executed when the filter catches a suspicious IP address. A jail can have a status of active or inactive. An inactive jail does not ban IP addresses. There are pre-configured jails for all Plesk services such as web server, ftp server etc. Jails can be managed from the Jails tab.

The Banned IP Addresses tab shows the list of IP addresses that have been banned by the Fail2Ban Jails. It allows un-banning IP addresses or marking them as trusted. The Trusted IP Addresses tab shows the list of IP addresses that will not be banned.

ModSecurity

ModSecurity is an open source cross platform Web Application Firewall. It checks all requests to the web server and responses from the web server against a set of rules.

If the check passes, the request is allowed. If it fails predefined action is taken. ModSecurity can be deployed either as a web server module or as a proxy server to a web server. It supports the major web servers which are Microsoft IIS, Apache and Nginx.

The Web Application Firewall (ModSecurity) for Plesk is available as a Plesk component. Once it has been installed, it needs to be enabled by going to Tools & Settings, then Web Application Firewall (ModSecurity).

Next set the Web application firewall mode to On. This will cause the Plesk ModSecurity to match each incoming request and its response against a set of rules. If the check succeeds, the request is allowed.

If it fails, the action is logged, a notification email is sent and the HTTP response is provided with an error status code. The Plesk ModSecurity mode can also be set to Off or to Detection only. When in detection mode, the action taken in case of failure is to simply log the request.

ModSecurity for Plesk uses the Atomic Basic ModSecurity rule set as the default set of rules for matching requests and responses. This can be changed to other rule sets such as OWASP ModSecurity, Advanced ModSecurity Rules by Atomicorp, Comodo ModSecurity or a custom rule set. Some of the rule sets require a subscription and are more strict and suitable for certain cases. The rule sets can be configured to update daily, weekly or monthly.

The rule sets can be configured to apply to the HTTP URI and parts of the HTTP header, the HTTP URI, header and HTTP POST request payload or the entire HTTP traffic including the HTTP response.

The ModSecurity audit log contains events logged by the ModSecurity. For example suspicious HTTP requests that fail to pass a rule. A security rule may be switched off, by entering the Security Rule ID, Tag or Regular Expression. This might be useful in case a HTTP request was incorrectly blocked. ModSecurity may be configured server wide for all domains, or for each domain.

Conclusion

Server security is an important task carried out by system administrators. Online applications are increasing in popularity. It is important that the servers that deliver these applications are properly secured.

Viewing package information using yum history command

Fri, 19 Oct 2018 00:00:00 GMT

The yum history command displays list of all commands used to install or remove packages using yum package manager. The list is ordered by date and id. Each history item is called a transaction.

To view details about a transaction enter the command:

yum history info transaction_id

where transaction_id is a number such as 15.

To view information about a certain package such as httpd, enter the command:

yum history info httpd

To view information about an entire package or list of packages for example httpd, use the command:

yum history package-list httpd or yum history package-info httpd

To undo changes made by a transaction, enter the command:

yum history undo transaction_id

To redo or reapply changes made by a transaction, enter the command:

yum history redo transaction_id

* To rollback or reverse changes made upto a certain transaction, enter the command:
```bash
yum history rollback transaction_id

See the article How to Use ‘Yum History’ to Find Out Installed or Removed Packages Info for more information on viewing package information using yum.

Installing Lets Encrypt SSL Certificates on Debian Jessie and Nginx

Sun, 14 Oct 2018 00:00:00 GMT

Introduction

HTTPS Certificates are files that are used to secure websites. A website that is secured using HTTPS is accessed using the HTTPS protocol. For example https://www.google.com. HTTPS certificates are issued by a Certificate Authority (CA). Lets Encrypt is a CA that issues HTTPS certificates free of cost.

Installation

To install a certificate from Lets Encrypt, we need to prove that we have control over the domain that needs to be secured. We can do this using an ACME (Automatic Certificate Management Environment) Protocol client. ACME Protocol is a protocol that allows automating Certificate Management Environments. Lets Encrypt provides several ACME clients. Certbot is the recommended ACME client which can be used to obtain HTTPS certificates from Lets Encrypt.

Certbot is packaged with Debian Jessie and can be obtained using the back-port repositories. To install Certbot on Debian Jessie, we can use the command:

sudo apt-get install certbot -t jessie-backports

This will install the Certbot client. To obtain a SSL certificate using Certbot, we need to enter the command:

sudo certbot certonly

This will allow us to interactively select the plugin and specify options that will be used to download the HTTPS certificate files. The webroot plugin is a simple plugin that downloads certificate files to the /etc/letsencrypt folder.

The webroot plugin can be used to obtain several certificates at a time. The plugin downloads verification files to the user specified folder. These files should be accessible over the internet. The Certbot server checks the presence of the files and then issues the requested certificates files. The files are saved to the /etc/letsencrypt folder. This folder contains several sub folders. The renew sub folder contains information used for renewing the certificate files. The live sub folder contains the certificate files used by the web server. It contains one sub folder for each domain.

Certificate Renewal

The certificates issued by Lets Encrypt are only valid for 90 days. After 90 days, the certificates must be renewed. Renewal of the certificates is very simple and simply requires the command:

sudo certbot renew

This will renew all certificates that were issued to the Certbot. The command can be run daily from cron. The Lets Encrypt installation guide describes the process of installing HTTPS certificate files using Certbot. The user guide describes the Certbot command line options in detail.

Certificate Removal

To remove a LetsEncrypt SSL certificate we can issue the command: certbot delete. It will display list of installed certificates, with option for deleting a certificate. When a certificate has been deleted, all the certificate related files are removed.

Conclusion

Lets Encrypt is an excellent service that allows securing websites free of cost. Their service is reliable and easy to use. It is also well documented and well supported.

Creating scalable applications with Ionic Framework and Firebase

Tue, 25 Sep 2018 00:00:00 GMT

Introduction

The Ionic Framework allows developing applications for web browsers, desktops and mobile devices. Ionic Framework applications are written in HTML, CSS and JavaScript. Apps can be generated from a single code base, which can be accessed from web browsers, Android, IOS and Windows based devices.

The Ionic Framework essentially is a front-end framework. The back-end for the apps can be implemented using Firebase or a server side programming language. The core concepts of the Ionic Framework describe how the framework works.

Difference between native, hybrid and mobile apps

The current version of the Ionic Framework, which is version 4 Beta supports several JavaScript frameworks in addition to AngularJs, such as ReactJs and Vue.

It essentially does not depend on any particular framework. It also uses Apache Cordova for generating hybrid apps from a JavaScript, HTML and CSS code base. Hybrid apps may be regarded as a hybrid between native apps and mobile apps.

Native apps are written in platform specific programming language such as Objective-C for IOS and Java for Android. Native apps have direct access to operating system libraries and tend to have better performance than hybrid and mobile apps.

Mobile apps are websites that are optimized for viewing on mobile devices. Hybrid apps on the other hand are written in platform independent JavaScript, HTML and CSS. Hybrid apps run inside a container called a Web View and have access by means of an Application Programming Interface (API) to device features such as camera, GPS, address book etc.

Installation

The Ionic Framework is a plugin for Node Js. It can be installed with the command: npm install -g ionic. The -g flag installs the package globally for all users. To enable support for developing hybrid apps, the Cordova NPM package needs to be installed with the command: npm install -g cordova.

Ionic Framework documentation describes how to get started with the Ionic Framework. To create an Ionic application, we need to enter the command: ionic start MyApp blank. This will create a new Ionic application in the current folder with the blank template. Supported templates are super, tutorial, blank, tabs and sidemenu.

Running Ionic Applications

The Ionic command line interface (CLI) has several options that can be viewed with: ionic help. To run the new app from a browser simply enter the command: ionic serve. This will start a web server that listens on port 8100. The command actually performs build tasks such as running Web Pack, trans-piling Type Script to JavaScript and converting SASS files to CSS.

Web Pack is used to bundle JavaScript files for use in browsers. Syntactically awesome style-sheets (SASS) is a script language that is compiled to Cascading Style Sheet (CSS). Ionic framework applications usually use SASS files for styling pages and Type Script for the front end programming. Converting code from one programming language to another is called trans-piling. For example converting Type Script to JavaScript.

If you have a production ready app then you may want to run it so the app uses compressed CSS and JavaScript and runs quickly. The following command line minifies the CSS and JavaScript files used by the app. It also disables live reloading of the app and causes the web server to run at port 9100. Live reloading allows an app to be modified without restarting the web server.

--mifyjs --minifycss --optimizejs --no-livereload --port=9100

Folder structure of an Ionic App

The ./src folder contains the source code for our application. It has a file index.html which is the main entry point for our app. We do not need to edit this file. The files we need to edit are the Type Script files that end with “.ts”. For example the file:src/app/app.module.ts is the entry point for the app. This file defines the root module which controls the application. The following example code defines the root module:

@NgModule({
  declarations: [MyApp, HelloIonicPage, ItemDetailsPage, ListPage],
  imports: [BrowserModule, IonicModule.forRoot(MyApp)],
  bootstrap: [IonicApp],
  entryComponents: [MyApp, HelloIonicPage, ItemDetailsPage, ListPage],
  providers: []
})
export class AppModule {}

The file ./src/app/app.component.ts declares the root module and the HTML template where the root module will be rendered.

When we run ionic serve, it creates a www directory containing the trans-piled JavaScript and CSS files. The contents of the www directory are served while the app is running.

Firebase

Firebase is a platform provided by Google, which allows scaling apps to billions of users. It essentially provides a set of services for developing the back-end of mobile apps. It supports major mobile platforms: Android, IOS, Web as well as the Unity and C++ platforms.

Firebase provides the following services:

Cloud Firestore. Provides a NoSQL database for storing data.
ML Kit. Provides APIs for common Machine Learning (ML) related tasks such as recognizing text, detecting faces, scanning barcodes, labeling images and recognizing landmarks
Cloud Functions. Allows code to be executed in response to events such as new user signup, conversion related events in analytics, changes to data in real time database etc. The user specifies the code to execute in response to events. The code is run as single purpose JavaScript functions in a secure Node Js environment. The user does not have to bother with server provisioning
Authentication. Firebase Authentication allows easily building secure authentication systems. It provides a flexible Drop-In user interface for adding authentication related features. It supports authentication providers such as email/password, Google, Twitter, Facebook, GitHub, Phone Auth and more.

Firebase provides several other services such as Hosting, Cloud Storage, Realtime Database, Crashlytics, Performance Monitoring, Test Lab, In-App Messaging, Google Analytics, Predictions, A/B Testing, Cloud Messaging, Remote Config, Dynamic Links and App Indexing

Conclusion

The Ionic Framework is a useful platform for building cross platform applications. Combined with Firebase, it allows developing scalable apps easily and quickly.

Increasing web server timeout

Tue, 25 Sep 2018 00:00:00 GMT

If your website is based on PHP and is being served by Apache web server with Nginx as reverse proxy, then you may have experienced 504 Gateway Timeout errors. The error usually happens because some PHP script takes longer than usual to run. If you want to allow the PHP script to run for more than the default time of 30 seconds, then you have to make some changes to the Php.ini configuration and the nginx.conf and Nginx virtual host configuration.

To increase timeout of PHP scripts, we need to increase the max_execution_time settings in php.ini. The default value is 30 seconds. The changes will take affect after the Apache web server has been restarted

The timeout also has to be configured on the Nginx reverse proxy. It requires disabling the keepalive probes sent by Nginx to Apache. This can be done by adding the ”Connection: ” HTTP header with empty value. For example:

proxy_set_header Connection "";

This needs to be set under the location block of the virtual host file located under /etc/nginx/sites-available

The fastcgi read timeout and proxy read timeout settings also need to be increased in the Nginx configuration. If you are trying to upload large files, then you need to increase the client_max_body_size setting. This can be done by adding the following:

fastcgi_read_timeout 600;
proxy_read_timeout 600;
client_max_body_size 20M;

to /etc/nginx/nginx.conf. After making the changes, nginx needs to be restarted.

See the discussion Nginx reverse proxy causing 504 Gateway Timeout for more information on how to increase web server timeout

Publishing websites using Jekyll

Fri, 07 Sep 2018 00:00:00 GMT

Introduction

Jekyll is a static site generator. It generates static html pages from content written in Mark Down or HTML. Its different from other website publishing tools in that it does not use a database.

The user writes his content in text files using HTML or Mark Down. Jekyll converts these files to html. The benefit of this approach is that it avoids the security and performance related problems of database driven websites. In this article I will describe my experience with installing “Crisp DNA”, a website based on Jekyll.

Installation

Jekyll is based on the Ruby programming language. It is actually distributed as a Ruby Gem, which means it can be installed easily using the gem install command.

Installing Ruby

Ruby is an interpreted language with a focus on simplicity and productivity. It is similar to the Python language, but unlike Python is used mainly for developing websites. Ruby may be installed using the Ruby Version Manager (RVM). RVM allows installing multiple version of Ruby each with its own set of Ruby Gems.

To install RVM, we need to first install the package signing keys using the command:

gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB

Next we need to install RVM using the command:

curl -sSL https://get.rvm.io | bash

Once RVM has been installed, we can use it to install Ruby. For example to list all the available versions of Ruby we can use the command:

rvm list known

To install a specific version of ruby, for example version 2.5.1, we can use the command:

rvm install ruby-2.5.1

Installing Crisp DNA

Crisp DNA is a consulting company. It has released the source code of its website on Git Hub under the Creative Commons License. The website of Crisp DNA is based on Jekyll. The website provides a detailed description on how the Crisp DNA company works. The website text is provided in the form of Mark Down files.

To install the website locally, we first need to download the website from Git Hub. Next we need to install the packages that are required to run the website.

We can do this with the commands:

gem install bundle

followed by:

bundle install

Next we need to run the command:

jekyll serve

from the root folder of the website. Alternately, we may run the ./run script which is provided with the website. This will start the Jekyll server on the localhost port 4000.

Structure of a Jekyll website

The installation and configuration of Jekyll is well documented on the Jekyll quick start quide. A website based on Jekyll has the following main components:

_config.yml. This is a configuration file for the website. It supports many options. Some of the useful settings are the host, port and baseurl.
_includes. This a folder that contains include files such as header, footer, sidebar etc.
_layouts. This a folder that contains the layout files for the website. A layout file is a base file which contains sections such as header, footer, main content, sidebar etc. A website may have several layout files.
posts. This is a folder that contains files with user content. The content may be formatted using HTML or Mark Down. Mark Down is a light weight markup language which is easy to use. A collection is a group of related files. The posts collection is a default collection. The user may create his own collections, which should be listed in the _config.yml file.
_site. This folder contains the static HTML files. These files are auto generated by Jekyll. One of the useful features of Jekyll is that user content may be modified while Jekyll is running. The Jekyll server does not have to be restarted each time a file needs to be changed.

A new Jekyll website may be created with the command: jekyll new website_name. This command will create the default folder structure. The folder structure is well documented on the Jekyll website.

Since Jekyll version 3.2, a default theme called minima is provided which provides a default theme for new websites. To use this theme we need to add the lines:gem “minima” to the Gemfile of the website and the line theme: minima to the _config.yml file.

Jekyll Admin

The Jekyll Admin is a useful web based tool that allows editing Jekyll website files. The Jekyll Admin is a Ruby Gem. To install the Jekyll Admin, we just need to add the lines:

gem 'jekyll-admin', group: :jekyll_plugins

to the Gemfile used by the Jekyll website. Next we need to start the Jekyll server. The Jekyll Admin should then be accessible on the URL: http://localhost:4000/admin.

Jekyll Plugins

The Jekyll plugin system allows us to add functionality to a Jekyll website without having to edit the Jekyll source code.

The plugins can be grouped as Generators, Converters, Commands, Tags, Filters and Hooks. For example the Jekyll SEO Tag plugin allows us to perform Search Engine Optimization (SEO) related tasks on our website.

Jekyll also provides import tools for transferring content from popular content management platforms such as WordPress, Blogger, Drupal etc to Jekyll.

Conclusion

Jekyll provides a unique approach to serving website content. It is a popular tool supported by a large open source community.

Getting /etc/rc.local file to work on Debian 9 (Stretch)

Wed, 05 Sep 2018 00:00:00 GMT

Debian Stretch uses the rc.local service for executing commands on system startup. To use this service we need to run the following commands:

If the /etc/systemd/system/rc-local.service does not exist, then it should be created with following contents:

[Unit]
 Description=/etc/rc.local Compatibility
 ConditionPathExists=/etc/rc.local

[Service]
 Type=forking
 ExecStart=/etc/rc.local start
 TimeoutSec=0
 StandardOutput=tty
 RemainAfterExit=yes
 SysVStartPriority=99

[Install]
 WantedBy=multi-user.target

Give executable permissions to the /etc/systemd/system/rc-local.service file

sudo chmod +x /etc/systemd/system/rc-local.service

Gve executable permissions to the /etc/rc.local file

sudo chmod +x /etc/rc.local

Enable the rc-local service to start on boot

sudo systemctl enable rc-local

Start the rc-local service

sudo systemctl start rc-local.service

Check the status of the rc-local service

sudo systemctl status rc-local.service

If the rc.local file does not exit, then create it with the following contents:

#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

exit 0

See the discussion How to Enable /etc/rc.local with Systemd for more information on getting /etc/rc.local to work on Debian 9

Making HTTP POST requests from Google Chrome

Tue, 14 Aug 2018 00:00:00 GMT

The Google Chrome developers toolbox is a useful tool that can be used to test HTTP POST requests. For example we may want to replay HTTP requests for testing APIS or debugging web pages.

To replay a HTTP GET or POST request in Google Chrome, we need to open the Chrome developers toolbox and then under the network panel right click on the URL to replay. It will give us the option to copy the request as a CURL command or simply replay the request.

If your URL request is over HTTPS, then your CURL command may give errors related to untrusted SSL certificates. The solution is to append the —insecure option to the CURL command.

See the official Chrome Developers Tool documentation for more information about making HTTP POST requests.

Using substr_compare function to find out if a file exists in a folder

Sat, 04 Aug 2018 00:00:00 GMT

The substr_compare function is a useful function that can be used to check if a given string occurs at the end of a larger string.

We can use the substr_compare function to check if a given file name segment occurs at the end of a given path. It can be used to implement a function that checks recursively if a file segment is present in a folder

The offset argument of the substr_compare function is set to the length of the file to search multiplied by -1. The length parameter is set to the length of the file name segment to be searched. The following code can be used. It checks if a string occurs at the end of a given string:

/** The file name segment is prefixed with '/' */
$file_name         = "/" . ltrim($file_name, "/");
/** The length of the file name */
$file_name_length  = strlen($file_name);
/** The offset at which to start the comparison. It is negative, so it is taken to be from the end of the string */
$offset            = (-1 * $file_name_length);   
/** If the file name segment occurs at the end of the item path */
if (substr_compare($item_path, $file_name, $offset, $file_name_length) === 0) {
    /** The function returns true */
    return true;
}
/** The function returns false */
else return false;

Another option is to simply split the string into two parts using explode function and then check if the second array element matches the file name segment. The following code may be used:

/** The larger string is split on the file name */
$path_segments = explode($file_name, $item_path);
/** If the second array element matches the file name, then the function returns true */
if ($path_segments[1] == $file_name) return true;
else return false;

Installing latest version of Php

Thu, 28 Jun 2018 00:00:00 GMT

PHP is a popular programming language commonly used for building web based applications. Most Linux distributions provide PHP out of the box. However the version of PHP in most cases is not the current stable version.

To install the latest stable version of PHP, we need to use a third part package repository such as deb.sury.org/. To use this repository we need to follow the instructions given on: https://packages.sury.org/php/README.txt.

After that we can install the latest version of PHP. For example to install PHP 7.2, we need to issue the commands:

apt-get update

followed by:

apt-get install php7.2

To revert to the version of PHP provided by the operating system, we need to remove the file: /etc/apt/sources.list.d/php.list. We also need to remove the package singing key for deb.sury. Next we need to run the commands:

apt-get update

followed by:

apt-get remove php7.2

After that we need to install the PHP package using:

apt-get install php

This will install the version of PHP that is provided by the operating system

Preventing spam messages with Contact Form 7 WordPress plugin

Thu, 28 Jun 2018 00:00:00 GMT

Contact Form 7 is a plugin for WordPress that allows website visitors to contact website owners. It is one of the most popular WordPress plugins with over 5 million active installations.

Different methods can be used to prevent spam messages from being sent through Contact 7 Forms. We need to first think about how spam is sent though the Contact 7 forms.

Spam may be sent by an automated script or bot. This type of spam can be stopped using captcha. Fortunately Contact Form 7 has built in support for captcha. It allows users to easily insert captcha in forms.

Another source of spam is human visitors that send junk email for different reasons. This type of spam may be stopped with the akismet configuration that is provided by Contact Form 7.

See the article 6 Ways to Stop Contact Form 7 Spam in WordPress – Updated for 2018 for more information on how to prevent spam messages with Contact Form 7.

Upgrading R from version 3.0 to 3.5

Thu, 28 Jun 2018 00:00:00 GMT

R is a free software package for statistical computing. It is part of most Linux distributions. However the latest version of R is not always available. To install the latest version of R we need to add the URL of a CRAN mirror to a file in /etc/apt/sources.list.d.

CRAN is a network of ftp and web servers that provide upto date information on R such as documentation, source code and news about R development. To install R, we need to issue the commands:

apt-get update

followed by:

apt-get install r-base r-base-dev

To update R to a newer version we need to replace the CRAN URL in /etc/apt/sources.list.d, with a URL that points to the newer version of R. After that we need to issue the commands:

apt-get update

followed by:

apt-get upgrade

Since the new CRAN URL requires different package signing keys, we will get an error similar to: “following signatures couldn’t be verified because the public key is not available: NO_PUBKEY”.

The solution is to import the package signing keys for the new R package using the command:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys [key_name]

where key_name is the short signature of the missing key as indicated in the error message. After that the apt-get upgrade command should work without problems.

See the R official documentation for more information.

Upgrading MySQL server from version 5.6 to 8.0

Thu, 28 Jun 2018 00:00:00 GMT

Introduction

The latest version of MySQL server currently is 8.03. To upgrade MySQL server from a previous version we need to ensure that we are upgrading from the last stable release.

For example from version 5.7 to 8.0. To upgrade from 5.6 to 8.0 we need to first upgrade from 5.6 to 5.7 and then from 5.7 to 8.0. Upgrading directly from 5.6 to 8.0, will result in errors related to incompatible SQL or log file formats.

Installing latest version of MySQL server

MySQL server is part of most Linux distributions. However the default version of MySQL server is usually outdated. We can install the latest version of MySQL server from the MySQL apt repository. The MySQL online documentation provides detailed information on how to install and upgrade MySQL server and tools using the MySQL apt repository.

To install the MySQL apt repository we need to first download the repository package from: http://dev.mysql.com/downloads/repo/apt/.

Next we need to install the repository package using the command:

sudo dpkg -i mysql-apt-config_w.x.y-z_all.deb

During the installation of the package, we will be asked to choose the version of MySQL server and other tools. After the repository package has been installed we can install the MySQL server using the commands:

apt-get update

followed by:

apt-get install mysql-server

This will install the version of the MySQL server that we had selected during the repository package installation. To change the version of MySQL server that we have installed, we need to issue the command:

sudo dpkg-reconfigure mysql-apt-config

The version that we select here will be installed when we install the package from apt.

Upgrading MySQL server

As mentioned previously we should upgrade to the next stable release. For example from 5.6 to 5.7. To upgrade MySQL server we should first back up the database. We can do this by exporting the databases to SQL file using the command:

mysqldump -u root -p --all-databases > database_file.sql

This command will export all databases to a SQL file. We may also back up the data directory located at: /var/lib/mysql.

Next we need to remove the MySQL server using the command:

apt-get remove mysql-server

This command will remove the MySQL server binaries from the system. To remove the MySQL data we need to substitute remove with purge in the previous command.

After that we need to choose the new version of MySQL server from the MySQl apt configuration tool. We can access the tool with the command:

sudo dpkg-reconfigure mysql-apt-config

Next we need to install the new version of the MySQL server with the command:

apt-get install mysql-server

During installation we will be prompted for the root password for the MySQL server.

If we had used the purge option while removing the MySQL server, then we will need to import the data back to the MySQL server. We can do this with the command:

mysql -u root -p > database_file.sql

After the installation has completed we need to upgrade the server using the command:

mysql_upgrade -u root -p

This will upgrade the table formats to the version support by the new MySQL server.

Downgrading MySQL server

To downgrade MySQL server to a previous version, we need to follow the same procedure that we used for upgrading the server. The only change would be to select the older MySQL server version from the MySQL apt configuration tool.

See the official MySQL documentation for more information.

Upgrading from Debian Jessie to Debian Stretch

Thu, 28 Jun 2018 00:00:00 GMT

Introduction

Debian 9, also known as Debian Stretch is currently the latest stable version of the Debian operating system. It was released on 17th June 2017 as successor to Debian 8 also known as Debian Jessie.

Under the Debian release cycle, stable versions of the Debian operating system are released every two years. Updates are provided for a total of three years. The Debian Security Team is responsible for providing security updates for the Debian operating system. Once a release has reached its end of life, the Debian LTS (Long Term Support) team takes over the job of providing security updates.

Debian Jessie which was released on 26th April 2017, reached its end of life on 17th June 2018. While it is still possible to receive security updates after this period, it is recommended to upgrade to the next stable release which is Debian Stretch. Google Cloud support sent notifications to its customers asking them to upgrade to Debian Stretch as soon as possible.

Debian Stretch contains numerous bug fixes as well as new updated versions of popular programming languages and servers. Upgrading to Debian Stretch is highly recommended. In this article I will describe my experience with upgrading the operating system of a server based in Google Cloud from Debian Jessie to Debian Stretch. I learned a few important lessons which should benefit system administrators.

Performing the upgrade

The process for upgrading from Debian Jessie to Debian Stretch is described in the Debian release notes. Although the upgrade guide is very detailed, the main steps are as follows:

Backup data and configuration information

You should ensure that all useful data on your server is backed up before starting the upgrade.

Inform users in advance and prepare for downtime on services

You should inform the users of your system about the downtime. Also prepare for downtime of the services running on the system, such as intranet websites.

Update the operating system packages on the current system

This requires installing updates to packages and removing unused packages. This can be accomplished with the commands: apt-get update followed by apt-get upgrade. After running these two commands, all packages installed on your system through the Debian package manager will be updated to the latest version. You should also remove unused packages using the command: apt-get autoremove.

Update apt sources

The file /etc/apt/sources.list contains URLs of package repositories. The URLs should be updated so they point to the repositories containing Debian Stretch packages. Apt files for custom repositories which are in /etc/apt/sources.list.d should also be updated.

Perform minimal system upgrade

This step involves installing minimum updates. This step is needed because the full upgrade process installs new version of all packages and also removes obsolete packages. It can sometimes remove useful packages.

It is therefore recommended to first perform a minimal upgrade before performing the full upgrade. The minimal upgrade can be performed using the commands:

apt-get update
apt-get upgrade
apt-get autoremove

Perform the full upgrade

The full upgrade installs new versions of all packages. It also resolves dependencies between packages and may install new required packages and remove obsolete packages.

The full upgrade can be done using the command:

apt-get dist-upgrade

While installing new versions of packages, it may prompt the user to choose whether to overwrite existing configuration files with newer versions or to keep the current version.

It may also provide an option for checking difference between new and current versions of the configuration files. After the full upgrade has completed, obsolete packages should be removed using the command:

apt-get autoremove

followed by a system reboot.

Problems with updating packages

The upgrade process described in the Debian documentation is easy to follow and should complete without issues. I was able to complete the operating system upgrade without problems. However I ran into issues when updating certain packages.

The Debian package repository usually provides old versions of programming languages and applications which are stable and tested for Debian but are not the current stable version.

The server I was upgrading had the following packages installed through custom repositories: R, PHP, MySQL and Postgresql. I thought it would be a good idea to upgrade these packages to the latest version.

Unfortunately, I did not consider the possibility that the applications that depend on these packages may not work with the newer package versions.

For instance the web based email management application, Mautic does not work with PHP 7.2. Also LimeSurvey does not seem to support MySQL 8.0. As a result several applications stopped working after the upgrade.

Conclusion

The saying: ”Dont fix what is not broken”, is a useful reminder to system administrators. Updating packages may seem like a good idea, but its consequences should first be determined, since it can cause important applications to malfunction.

Improving WordPress Performance

Sat, 23 Jun 2018 00:00:00 GMT

Introduction

Recently I had the opportunity of working on improving the performance of a WordPress website for a customer. I will describe my experience in this blog post. Slow website loading times can distract your website visitors and discourage them from visiting your website again.

It is best to conduct a thorough performance tuning of your website before it is launched to the public. You can create a development version of your website and test it for performance. After it has been tested you can make the website live.

The popularity of WordPress

WordPress is a very popular platform for hosting different types of websites. According to the article, What Code You Should Learn, over 19% of all websites are based on WordPress. WordPress has over 25,000 plugins and themes that perform just about any task you can expect of a website. 47 WordPress Statistics We Bet You Didn’t Know in 2020 has some interesting statistics about WordPress.

Improving WordPress performance

There are many factors that can slow down your WordPress website. Knowing exactly what is slowing down your website can be difficult. Fortunately there are several free on-line tools that can help you troubleshoot website performance problems.

Process for tuning WordPress performance

The process I used for tuning the performance of a WordPress website is very simple and can be described as follows. You need to run one or more on-line tools that analyze the performance of your website. These tools provide recommendations on how to improve your website. Implement these recommendations and run the tools again. Then implement the recommendations of these tools again. Continue doing this until the tool gives a good performance figure for your website.

Tools for performance tuning

Some of the tools used for performance tuning are:

Dotcom monitor

This is a useful service that checks your website in real web browsers in over 20 locations. It gives average load time and waterfall of your website for each location as well as an overall average loading time and waterfall for your website.

A waterfall is a chart that provides detailed information about the resources on your website. For example it gives the size and loading time of each CSS, JavaScript and image file. It helps you to pinpoint the exact resource that is slowing down your website.

With this tool you can also find out how much time your website takes to load for visitors from certain countries. The tool allows you to select the type of browser to be used for testing. Currently it supports Google Chrome, Firefox, Internet Explorer versions 7 and above, Iphone, Ipad, Ipod, IpadMini.
Google webmasters mobile friendly tool

This is a useful service that gives a measure of mobile friendliness of your website. The tool also provides reasons why your website is not mobile friendly. For example “Text is too small to read”, “Links too close together”, “Mobile view port not set”. Mobile friendliness is now an important criterion in Google search rankings. This was recently announced by Google in the article “Get Ready For Google’s New Mobile Ranking Signal”
Google Pagespeed

This is a useful service that runs Google Pagespeed tool on your website. The tool describes how your website runs on mobile devices and desktops.

For mobile devices, the tool provides a score out of 100 for User Experience and Speed. It also highlights the areas in which your website is lacking and gives recommendations. The tool also gives an overall Pagespeed score out of 100 for Desktop devices.
GTmetrix

This is a very useful service that provides a Performance Report of your website. It gives following information: Page Speed Grade, Y Slow Grade, Page load time, Total page size, Total number of requests and a Breakdown. The Breakdown gives recommendations on how to improve the page load time.

It basically just gives the recommendation of the Google Page Speed tools and YSlow. You can also export the performance report as PDF.
Pingdom Website Speed Test

This is an excellent tool that gives a detailed description on the requests done to load a web page. For each request made for downloading a resource such as a CSS file, the tool gives detailed information such as time taken to load the resource including breakdown of the time taken and the size of the resource. The breakdown of the time taken includes time taken for DNS, Connect, Send, Wait and Receive and Total.

The main features of the tool are a performance grade out of 100, number of requests made by the browser, page load time and page size. The tool also gives detailed information about the Waterfall, Performance grade, Page analysis and History.

The level of information provided by all these tools is amazing. Although all tools provide similar information, it is good to run your website on all the tools so you get an overall measure of your website’s performance

Conclusion

Ensuring good performance of your website is very important, especially if your website is the public image of a company and is visited by many visitors.

Ensuring good performance of a website requires thorough research. Website performance improvement is a useful skill to learn that is well worth the effort.

Bad performance of virtio network drivers on Proxmox

Sun, 20 May 2018 00:00:00 GMT

Introduction

This blog post is about my experience with trying to optimize network performance on a virtual machine managed by the Proxmox virtualization platform. I described my experience with using Proxmox in an earlier article titled Proxmox - An open source Virtualization Platform.

The conclusion of the article was that Proxmox is an excellent virtualization platform which supports LXC, OpenVZ and KVM virtual machines. It can be used to host a cloud platform delivering all kinds of services. Proxmox has an excellent user interface that allows managing virtual machines. Proxmox is very easy to install and administer. It also has an excellent support forum where you can get help on Proxmox related issues.

Problems with accessing websites

The issue was that I had installed a database management script called PhpMyAdmin on a Proxmox KVM virtual machine. I was not able to access the script from a web browser. Finding the cause of the issue took me several days!.

Possible issues

I had to consider a lot of issues. SELinux, Apache web server configuration, Gateway NAT configuration and TCP MTU settings on all the network devices. I even tested the bandwidth on the virtual machine. All those factors could have caused the issue. The most obvious cause of the issue seemed to be TCP MTU settings on a network device.

Problems with MTU

We had updated MTU settings on some of our servers with the hope of improving network performance. TCP MTU is the maximum size of data that can be transmitted in a single TCP/IP packet. TCP MTU is a very basic network parameter that should not be changed unless you know what you are doing. I thought that had to be the cause of the issue. I reset the MTU to default value on all the network servers, virtual machines, switch and gateway. But that did not help !. I was still getting a blank page when trying to access PhpMyAdmin from a browser.

The page would take time to load and would end up blank. I ran Wireshark on my local computer and checked the network packets. Some of the packets were being re-transmitted so there seemed to be a problem in the network. I then tested the bandwidth on the virtual machine using Iperf network performance testing tool. Iperf showed bandwidth of only 3Kbps!.

Solution: Problems with Virtio Network Interface Card

I then decided to try my luck with changing the network interface card type from Virtio to E1000. E1000 is a stable Intel network driver. Virtio gives the virtual machine access to the local host hardware and is supposed to be much faster then E1000. In this case it proved to be the opposite. As soon as I replaced the Virtio network card with E1000 the bandwidth on the virtual machine jumped up to 5Mbps !. And guess what the PhpMyAdmin started working !.

Conclusion

Changing the MTU on a network can make it difficult to troubleshoot network related problems. Its best to keep the MTU at its default value. Virtio network drivers usually have excellent performance. We were using VLANs with our virtual machine so that could have been a factor. I could not find the issue mentioned on the Proxmox forums so I reported it on Proxmox Bug tracking system. Its a good idea to contribute to the open source community, especially if you are a big user of open source software.

If you are interested in website monitoring applications, then checkout the Free Website Monitor App. It allows you to monitor the status of your websites from your phone.

Backing up and restoring LXD containers

Tue, 01 May 2018 00:00:00 GMT

LXD containers can be backed up by creating snapshots or by creating an image from the container. The first method which is creating snapshots is useful for taking quick incremental backups of containers. However it does not provide a full backup. The second method allows creating full backup.

To take a full backup of a container we need to run the following commands:

// generates an image from the container
sudo lxc publish {container_name} --force --alias {image_name}
// saves the image as a file. the file can be copied to secondary storage such as usb
sudo lxc image export {image_name} .

To restore the backup, we need to first manually import the image using the following command:

sudo lxc image import {file} --alias {image-name}

Next we need to launch a container from the image using the command:

sudo lxc launch {image-name} {container-name}

See the article How to properly backup+restore LXD containers with ZFS backend? for more information on how to backup and restore LXD container

Adding event listeners using JavaScript

Wed, 04 Apr 2018 00:00:00 GMT

The addEventListener function can be used to listen for events. The function can be called on any HTML DOM object such as HTML elements, HTML document, the window object or objects that support events such as xmlHttpRequest. The following example shows how to attach event listener to the window object:

window.addEventListener("resize", function(){
    document.getElementById("demo").innerHTML = sometext;
});

The above code sets the inner HTML of the HTML element with id ‘demo’ to some text. The first parameter of the addEventListener function is the name of the event to monitor. The second argument is the handler function. An optional third argument allows specifying the event propagation method.

Two event propagation methods are supported. “Bubbling” and “Capturing”. The Bubbling method implies that the events are handled from inside out. For example if a div tag contains a paragraph tag, and the paragraph tag is clicked, then the event handler for the paragraph tag will be called first and then the event handler for the div tag will be called.

In the “Capturing” method of event propagation, the events are handled from the outside in. In the previous example, if we click on the paragraph tag, the event handler for the div tag will be called first, followed by the event handler for the paragraph tag.

See the W3schools article JavaScript HTML DOM EventListener for more information

Creating custom playlists for MediaElementPlayer

Sat, 24 Mar 2018 00:00:00 GMT

Introduction

MediaElement Player is a HTML/CSS Video/Audio player for the web. It is based on MediaElement.js media framework library. Both player and library are available as open source and free of cost under the MIT license.

MediaElement Player allows playing audio and video from different sources such as YouTube, SoundCloud, DailyMotion, Facebook, MP4, FLV and more. The player is easy to setup and configure.

For browsers that do not support html5 or certain media codecs such as H.264, MediaElement Player provides a flash based UI. The flash based UI provides the required media codecs or the missing html5 features.

MediaElement Player supports IE11+, MS Edge, Chrome, Firefox, Safari, iOS 8+ and Android 4.0+.

Installation and configuration

To install MediaElement Player, we need to first clone the mediaelement git repository. After that we need to copy the contents of the build folder to the document root.

To create a sample page showing the player, we need to include following CSS file in the head section of the html page:

<link rel="stylesheet" type="text/css" href="mediaelementplayer.min.css">

We also need to include the following file just above the closing body tag:

<script src="mediaelement-and-player.min.js"></script>

Next, we need to instantiate the video player using the following code:

<div class="player">
  <video id="player1" width="640" height="360" preload="none"
     style="max-width: 100%"
     controls poster="images/big_buck_bunny.jpg" playsinline webkit-playsinline>
     <source src="http://clips.vorwaerts-gmbh.de/big_buck_bunny.mp4" type="video/mp4">
     <track src="dist/mediaelement.vtt" srclang="en" label="English" kind="subtitles" type="text/vtt">
  </video>
</div>

The above code will create a video player and load it with the a sample mp4 file with subtitles.

MediaElement Player plugins (VAST, VPAID, Postroll and Google Analytics)

The MediaElement.js provides several plugins. Some useful plugins include postroll, which allows displaying custom html at the end of the video and the ads plugin, which along with the vast/vpaid plugin allows displaying video ads.

VAST (Video Ad Serving Template) is a specification released by the Interactive Advertising Bureu (IAB), which describes the communication standard between video players and ad servers. For example, in order to play a video ad, the video player sends a request to an ad server, which returns the XML conforming to the VAST standard. The xml structure describes the details for displaying the video ads.

VPAID (Video player ad-serving interface definition) is a specification by the IAB which describes interactive communication between ad server and video player. It allows video players to display interactive video ads

The Google Analytics plugin allows recording video events such as play, pause and end. These events are recorded by Google Analytics and can be viewed from the Google Analytics console.

Creating custom playlist for the MediaElement Player

We needed a html5 video player with support for YouTube playlists. MediaElement Player was the best option, but its playlist plugin did not support YouTube videos. Also the playlist layout was not good.

Fortunately the source code for the playlist demo was easy to customize. The MediaElement Player API describes the different JavaScript functions for controlling the player and also player configuration options. I came across a couple of issues while using the player.

The first issue was that the player did not allow setting the YouTube configuration dynamically.

Second issue was a full screen error when using the playlist plugin. The error was that in the playlist demo sample, when the player is in full screen mode and the big play button is clicked, the player resizes without properly exiting the full screen mode.

We used the playlist plugin to develop our own YouTube playlist. See the YouTube Playlist article for details on how the YouTube playlist works.

We felt that the YouTube Playlist project could benefit from the open source community’s support. We therefore decided to publish the source code for the project on GitHub.

Conclusion

MediaElement Player is a useful video and audio player for the web. It is being actively developed and is supported by a large open source community

Displaying contents of an array in NXM html table

Sat, 24 Mar 2018 00:00:00 GMT

The following code allows displaying the contents of an array containing X number of elements in a HTML table. The html table contains N rows and M columns. The values of X, N and M can be configured:

<?php

/** The column counter */
$column_counter = 0;
/** The row counter */
$row_counter    = 0;
/** The total counter */
$total_counter  = 0;
/** The maximum number of columns to show */
$max_columns    = 3;
/** The maximum number of rows */
$max_rows       = 10;
/** A two dimensional array for storing the data */
$data           = array();

/** The sample data */
$model          = array_fill(0, 25, "test");

/** Convert the $model array into a two dimensional array */
foreach($model as $d) {
	/** The data is added to array */
	$data[$row_counter][$column_counter] = $d;

	/** The total counter is increased by 1 */
	$total_counter++;
	/** The row counter is increased by 1 */
	$row_counter++;

	/** If the total counter is divisible by $max_rows, then column counter is increased by 1 and row counter is set to 0 */
	if ($total_counter % $max_rows == 0) {
		$column_counter++;
		$row_counter=0;
	}
}

/** The start table tag */
echo "<table>";
/** Display the two dimensional data in html table. Each row is checked */
for ($count1 = 0; $count1 < $max_rows; $count1++) {
	/** The row start tag is displayed */
	echo "<tr>\n";
	for ($count2 = 0; $count2 < $max_columns; $count2++) {
		/** If the column does not exist, then empty column is shown */
		$column_data   = (isset($data[$count1][$count2])) ? $data[$count1][$count2] : " ";
		/** The column data is shown */
		echo "<td>" . $column_data . "</td>\n";
	}
	/** The row end tag is displayed */
	echo "</tr>";
}
/** The end table tag */
echo "</table>";

?>

See the discussion 3 column layout - how to have a limit in each column for more information about how to display contents of an array in a HTML table

Using Prefetch, Preload and Preconnect for speeding page load times

Mon, 19 Feb 2018 00:00:00 GMT

Prefetch, Preload and Preconnect are mechanisms supported by modern web browsers that allow efficient loading of web resources.

Preload allows controlling how resources such as images, CSS and JavaScript files are loaded on the current page. For each resource that we want to preload, we have to use a link tag with rel=“preload” attribute. The following examples show how to preload resources:

<!-- allows preloading an image -->
<link rel="preload" href="image.png">
<!-- allows preloading a font file -->
<link rel="preload" href="https://example.com/fonts/font.woff" as="font" crossorigin>
<!-- allows preloading css -->
<link rel="preload" href="https://blog.keycdn.com/blog/css/mystyles.css" as="style">

The main benefit of the preload directive is that it allows important resources to be loaded earlier. For example if there is a stylesheet or image that is required for a page to display properly, then it can be preloaded by adding the relevant link tags in the head section of the document.

Prefetch is similar to preload but allows efficient loading of resources that are likely to be accessed by the user, such as a next or previous link. A drawback of prefetch is that it can cause certain page metrics such as ad statistics, visitor count etc to be miscalculated. For example Google Chrome has been known to miscalculate web analytics stats. There are three types of prefetch directives:

link-prefetch. It allows a link to be fetched in advance before it has been clicked by the user. The contents of the link are saved to the browser cache
dns-prefetch. It allows the host name of the resource to be resolved in advance. It eliminates the dns resolution step when the resource is requested by the user
pre-render. It allows a resource to be fetched and processed before being requested by the user. For example a link marked with the directive rel=“prerender” will be fetched and processed by the browser. It should be used with caution as it can consume resources on mobile devices
preconnect. The preconnect directive allows establishing tcp connections for a resource before it is actually requested. For example tls negotiations, tcp handshake and dns resolutions.

See the article Resource Hints - What is Preload, Prefetch, and Preconnect? for more information on using Prefetch, Preload and Preconnect.

Securing Linux servers with SeLinux, ClamAV, SpamAssassin and RkHunter

Sun, 28 Jan 2018 00:00:00 GMT

Introduction

Securing your servers is a very important task that can save your organization a lot of problems. Recently there have been a lot of cases where insecure servers belonging to high profile organizations have been exploited and used for nefarious purpose. It is therefore important to secure your servers.

Recently I had the opportunity of securing Linux based servers using the tools SELinux, ClamAV, SpamAssassin and RkHunter.

Securing your Linux servers using SELinux

SELinux is a mandatory access control system for operating systems based on the Linux Kernel. It is implemented as a Linux kernel security module.

Mandatory access control vs Discretionary access control

Mandatory access control implies the system administrator enforces access control for resources. This is in contrast to Discretionary access control, where the access control is enforced by the owner of the resource.

Discretionary access control is the default in Linux. It involves assigning permissions on files using the commands chmod and chown. SELinux is applied after the normal discretionary access control. If access to a resource is denied by discretionary access control system, then SELinux is not even consulted.

Similarity with AppArmor

SELinux is similar to AppArmor, which is used by SuSe and Ubuntu operating systems. AppArmor is also implemented as a Linux kernel security module and provides mandatory access control but the main difference with SELinux is that it works with file system paths whereas SELinux works with Inodes.

SELinux uses extended attributes of file systems so it does not work on file systems that do not support extended attributes such as file systems mounted over NFS. Most users find SeLinux more difficult to administer. However SELinux offers more fine grained control and is included by default in the Linux Mainline Kernel.

SELinux Contexts

The term SELinux context is often referred to as SELinux label. The Gentoo Wiki has excellent information about SELinux. According to the wiki, SELinux uses contexts to identify resources. A context has 3 and sometimes 4 parts. These parts are:

** SELinux user**
SELinux role
SELinux type
SELinux sensitivity

The SELinux user and role is usually same for different uses and processes. In most cases the SELinux type differentiates the security context. It is sometimes referred to as the SELinux context. The SELinux sensitivity is also called Multi-level security. It provides classification of data. e.g strictly confidential data, confidential data, internal data and public data.

An example of a SELinux context is: user_u:user_r:user_t:s0. Contexts are applied to all system resources including sockets, ports, files and directories.

SELinux also adds a new option usually a -Z switch to resource monitoring tools such as ls, id, ps and stat. For example to view the SELinux context of the current user, use the command **id -Z. **To view the SELinux context of files use the command: ls -Z file_name.

The SELinux security system checks all resource access against a policy. If the policy disallows the access, then the access is denied and a message is logged to a file such as /var/log/audit.log.

Activating SELinux

SELinux is activated by default. To check if its activated use the command getenforce. To enable SELinux temporarily use the command setenforce 1. To permanently enable/disable SELinux edit the file /etc/selinux/config.

Many system administrators choose to disable SELinux because it causes problems with certain applications. I think this is a big mistake, because SELinux is not designed to block applications but to secure them. A basic understanding of SELinux can easily help to troubleshoot these problems.

Changing SELinux Context

The most common system administration task for SELinux is to allow a process access to a file or directory. e.g Apache web server has access to only files and directories with a certain SELinux context. If you want to allow Apache access to other files or directories then you need to update the SELinux context of those files and directories.

To update the SELinux context of a file or directory temporarily we can use the following command:

chcon -t net_conf_t /etc/puppet-resolv.conf

This command sets the context of the file puppet-resolv.conf to net_conf_t.

To make the change permanent we need to update the SELinux user space definition list and then relabel the files. We can update the user space definition list with the command:

semanage fcontext -a -t net_conf_t "/etc/puppet-resolv\.conf"

To relabel the list we use the command:

restorecon -R /etc/puppet-resolv.conf /usr/lib/portage/bin

This will permanently change the SELinux context of the file. Errors caused by SELinux can be seen in the file /var/log/audit.log.

The Centos Wiki also has some useful information on SELinux. According to the wiki we can use the setroubleshoot package for troubleshooting SELinux related issues. This package provides the sealert command which gives a user friendly version of the contents of the audit.log file.

ClamAV

ClamAV is a well known anti virus program that is available for several operating systems including Windows, Linux, BSD, AIX and Solaris. ClamAV contains the following components: command-line scanner, automatic database updater and a scalable multi-threaded server.

ClamAV also contains a Milter interface that allows applications that support the Milter extension such as Postfix and Sendmail to use the ClamAV for checking for viruses.

The ClamAV website has complete information on installing ClamAV. There are many on-line tutorials that describe how to use ClamAV. For example ”Automate ClamAV to Perform Daily System Scan and Send Email Notifications on Linux” and ”How to Install ClamAV and Configure Daily Scanning on CentOS“.

Rootkit Hunter

Rootkit Hunter or Rkhunter is a well known tool that scans for rootkits, backdoors and possible local exploits. It does this by comparing file signatures of well known files with signatures in an on-line database. If the signatures do not match then the Rkhunter reports an error.

Rkhunter installation is simple and is well documented on Installation Rootkit Hunter (rkhunter) on CentOS 5 and 6.

SpamAssassin

SpamAssassin is a well known spam filtering program that can be run on different operating systems including Linux and Windows and is compatible with most email servers. SpamAssassin can be run as a standalone server or integrated with email servers.

The SpamAssassin wiki has a useful guide for installing SpamAssassin. Installation of SpamAssassin is simple. On Centos it can be installed with the command yum install spamassassin..

Integrating SpamAssassin with Postfix

To integrate SpamAssassin with Postfix we need to first edit the file /etc/mail/spamassassin/local.cf and configure the score for spam messages, the text to append to the email subject, whether to move the email to the inbox or delete the email and the spam score for all email that is sent out by SpamAssassin.

After editing the configuration file, we need to add a system user for SpamAssassin and then configure Postfix so it uses SpamAssassin. After that we need to run a cron job that updates the SpamAssassin rules and reloads SpamAssassin.

Installation of SpamAssassin and integration with Postfix is described on Integrating SpamAssassin into Postfix using spamd, Integrating SpamAssassin into Postfix using spampd and SpamAssassin Integration with Postfix, using Amavis.

Integration with other Mail Transfer Agents is described on Integrated into several varieties of MTA. The article Installing and configuring Spamassassin on CentOS contains useful information on how to configure SpamAssassin with Postfix.

Conclusion

Securing your server is a very important task that should always be considered during server administration. A secure server can save you and your organization lots of problems.

Understanding Meltdown and Spectre vunerabilities

Sun, 28 Jan 2018 00:00:00 GMT

Introduction

Meltdown and Spectre are hardware vulnerabilities that affect all microprocessors based on Intel and some based on ARM and AMD.

The vulnerabilities have been under research since last several years but were made public in January 2018. Security experts consider the threats posed by these vulnerabilities as very severe.

Virtual Memory and Paging Tables

In order to understand the Meltdown and Spectre vulnerabilities, it is helpful to know technical terms such as Virtual Memory and Paging. Virtual Memory is a contiguous address space that is used by processes.

It works as an abstraction layer over physical memory. It gives each process, the impression that it is the only process running on the system. Even though each process has its own virtual memory space, the physical address space of processes may overlap.

Virtual Memory is mapped to physical memory using paging tables. It is maintained by the operating system. Depending on the operating system, all processes may share a paging table or each process can have its own paging table.

Data is read in fixed sized blocks called pages from I/O devices and loaded to memory. If a process needs access to data and there is not enough memory to store the data, then existing pages of data are written back to the I/O device and the required data is read and stored in memory. This is called swapping.

Thrashing is said to occur, when a process repeatedly swaps data in and out of memory because of insufficient physical memory. It can be prevented by adding more memory or by optimizing the application or operating system

The Meltdown vulnerability

Meltdown vulnerability allows a process to access memory location that is is not authorized to access. An example of an attack based on meltdown is as follows:

A rouge process uses indirect addressing to access a memory location. The instruction consists of a base value, for example 5000 and a memory address for example 2000. This address is out of bound to the process. The contents of this address is 4 and has size of 1 byte. The indirect instruction is therefore supposed to fetch the contents of memory location 5004.
When the CPU tries to fetch the contents of a memory location, it has to check whether the process should have access to this location. However this check which is known as a privilege check is performed in parallel with the memory fetch operation in order to speed up instruction execution.

As part of the memory fetch operation, the contents of the memory which is 4 is saved to the CPU cache. The next time address 5004 needs to be read, its value will be read and saved to cache before the privilege check completes. The read instruction itself fails and the process never gets direct access to memory location 2000 or the contents of the CPU cache.

The rouge process then uses a cache timing attack which is a form of side channel attack to determine that request for address 5004 takes less time to return as compared with requests for other addresses. The process can then conclude that memory location 2000 contains the value 4, even though the location is out of bounds to the process.

The Spectre Vulnerability

The Spectre vulnerability is based on exploiting the side affects of speculative execution, in particular branch prediction which is a special case of speculative execution.

Speculative execution is an optimization technique that is used to complete work before it is known that the work is needed or not. Branch prediction is used by the CPU to predict the outcome of a condition which results in execution of a certain block of code. Unlike Meltdown, Spectre is based on a more general concept and is more difficult to protect against.

Which devices can be affected?

The Meltdown and Spectre vulnerabilities affect all servers, desktops, laptops, mobiles and embedded devices. Cloud service providers are affected by Meltdown in a limited way.

The Meltdown vulnerability allows rouge processes running on the host to access guest virtual machines, but does not allow guest virtual machines to access other guests or the host.

Web Browsers that use Just In Time (JIT) JavaScript compilation which include all modern web browsers can be affected by Meltdown and Spectre. For example Meltdown can be used by a website to determine the information of another website that has been loaded by a user in a different tab.

Steps taken to mitigate the affects of Meltdown and Spectre

Kernel Page Table Isolation (KPTI) is a measure taken to reduce the impact of Meltdown. It involves isolating kernel memory from user mode processes. KPTI patches have been developed for the Linux Kernel 4.15 and also for older versions of the kernel. These patches have been merged with the main kernel.

The KPTI patches have been developed by all major Linux vendors including RedHat. Apple and Microsoft have also released security updates containing KPTI patches.

The KPTI patches are known to reduce computer performance. However the performance impact affects certain workloads more than others. For example the performance of I/O intensive applications such as PostGres database server is more affected by KPTI patches.

Conclusion

The Meltdown and Spectre vulnerabilities are serious threats which affect almost all microprocessors. No known threats based on these vulnerabilities have yet been discovered. Given the severity of the threats, attacks based on Meltdown and Spectre may soon surface.

Proof of concept code which demonstrates Meltdown and Spectre is already available publicly. Attacks based on Meltdown cannot be detected. Attacks based on Spectre are even more difficult to predict and affect a wider range of devices.

Installing Laravel Spark, GitScrum and IceCoder on Debian Stetch

Sat, 13 Jan 2018 00:00:00 GMT

Introduction

Laravel Spark is a non-free but open source application for the Laravel PHP Framework. It provides features that are commonly found in web applications such as user subscription, billing plans, invoicing and payment processing.

Laravel GitScrum is a free and open source application that provides graphical statistics on GitLab accounts. It allows synchronizing project and issue information with GitLab.

In this article, I will describe my experience with Installing and configuring Laravel Spark and Laravel GitScrum. I will also describe how to add a custom theme to the Laravel Framework

Installing Laravel Spark

Laravel Spark installation is well documented on the Laravel Spark website. The main pre-requisites for the installation are: Composer, NodeJS & NPM. Composer is a packager manager for Php, while NPM is a package manager for NodeJs. NodeJs is a server side JavaScript runtime environment which allows Javascript code to be executed on the server.

Installing NodeJs and NPM

NodeJs and NPM can be installed using the Node Version Management (NVM) script. The script can be downloaded from its Git Hub repository. NVM can be installed using the command:

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | bash

This will install NVM in the home directory. The install location can be customized using the NVM_DIR environment variable.

Once NVM has been installed, we can use it for installing our required version of NodeJs and NPM. NVM allows installing multiple versions of NodeJS. For example to install the latest version of NodeJs, we can use the command:

nvm install node

To check the version of NodeJs we can use the command:

node -v

Once the pre-requisites have been installed, we can start installing Laravel Spark. The command:

composer global require laravel/installer

will install Laravel globally using Composer.

Next we need to clone the laravel/spark-installer repository from Git Hub. Once the repository has cloned, we need to install the dependencies for Laravel Spark using the command:

composer install

The command should be issued from the root of the cloned repository. The cloned repository path should be added to the $PATH environment variable, so the spark command can be run from any folder on the system

Next we need to register our application with the Spark installer. To do this, we need to generate an API token from our Spark account. We can then register our API token using the command:

spark register token-value

where token-value is the generated token

We can now create new Laravel Spark projects by issuing the command:

spark new project-name

where project-name is the name of the folder where the Spark application should be installed. Once we have created a new Spark project, we need to migrate the database for the new project using the command:

php artisan migrate

This will create new database tables. Before running this command, we need to enter our database credentials in .env file.

We should then be able to access the Laravel Spark frontend page at the url configured in the virtual host configuration file. The frontend page has two buttons for logging in and registering. When the user logs in, he is taken to his application dashboard. He will also be able to view his personal information and subscription information.

Laravel Spark supports the Stripe and BrainTree payment processors

Installing Laravel GitScrum

Laravel GitScrum is a free open source application that essentially works as an agile frontend for the GitLab API. It allows importing projects and issues from GitLab.

It also displays charts and graphs describing project and issue progress. It provides product backlog, sprint backlog and user stories. GitScrum requires Php 7.1 or higher. It can use MySQL or Mariadb as the database server.

GitScrum installation is well documented on the GitScrum repository front page. To install GitScrum, we need to first clone the GitScrum source code repository by issuing the command:

git clone git https://github.com/gitscrum-community-edition/laravel-gitscrum.git

Next we need to install the GitScrum dependencies using the commands:

cd laravel-gitscrum && composer update
composer run-script post-root-package-install

Next we need to set the Application URL by editing the constant APP_URL in .env file. The database connection information should also be added to the .env file.

Next we need to create the database tables for GitScrum by running the commands:

php artisan migrate && php artisan db:seed --class=SettingSeeder

Next we need to enter our GitLab key and secret in .env file. To fetch the key and secret values, we need to first create a new GitLab app and fetch the application Id and Secret.

Installing IceCoder

IceCoder is a simple web based code editor. It can be used to edit source code files on the server. It has a user friendly interface and can be configured to display files in any folder, provided the user under which the web server is running has permissions to read/write the files. It only requires PHP and does not use a database.

To install IceCoder, we simple need to download the files from the IceCoder website. Once the files have been downloaded, we need to extract them to the document root of the virtual host. The extracted files should be owned by the user under which the web server is running. We should then be able to access the IceCoder from browser by navigating to the server URL given in the server virtual host configuration.

We can configure the root folder that will be served by the IceCoder. This can be done by setting the docRoot configuration in lib/config___settings.php. By default it is set to the virtual host document root.

The IceCoder also supports viewing and editing files remotely such as over ftp or GitHub.

Adding template files to Laravel Framework

The Laravel Framework has a built in template engine called Blade. It allows html templates to be easily added to the Laravel Framework.

To quickly add a template file to Laravel framework, we need to first add a route over which the file will be served. We can do this by adding the following to routes/web.php.

Route::get('/', function () {
    return view('/custom/home1');
});

This will cause the file home1.blade.php inside the folder views/custom/ to be served.

Blade template files need to end with .blade.php. They contain plain HTML but can optionally contain configuration directives. For example the statement: @extends(‘layouts.main’) will cause the current view file to use the layout file: layouts/main.blade.php. The directive: @section(‘title’, ‘Welcome to ValueHub’) will case the page title to be set to: ‘Welcome to ValueHub’.

The template file can also contain sections with content that is auto inserted into other template files. For example the main layout file layouts/main.blade.php can contain the code: @yied(‘content’). Each view file that contains content should have code which starts with @section(‘content’) and end with: @stop.

Template files can also include other template files. For example the command:

@include('custom.header')

will cause the template file: views/custom/header.blade.php to be included. This can be useful for including a header or footer template file.

Conclusion

Laravel Spark provides the scaffolding for developing applications that require subscription and payment processing. It a useful starting point for developing applications.

GitScrum provides tools for developing applications according to agile methodology. It provides Burn Down charts, Backlogs and Kanban boards for managing application development. Both applications can be used in the development of custom applications.

Using cookies in Javascript

Sat, 16 Dec 2017 00:00:00 GMT

Cookies allow websites to store information on the user browsers. They are usually used to store website session related information. To set a cookie using JavaScript we can use:

document.cookie="cookie_name=cookie_value";

Multiple cookie values should be separated by ’;‘.

The cookie string can contain the date of expiry of the cookie as well as the URL path for which the cookie is valid. By default the cookie belongs to the current page. An example of setting a cookie using JavaScript:

document.cookie = "username=John Doe; expires=Thu, 18 Dec 2013 12:00:00 UTC; path=/";

To fetch the value of a cookie we can assign the document.cookie property to a variable. For example: var cookies = document.cookie. We will then have to parse the values of the individual cookies

To delete a cookie, we need to set the value of the cookie to an empty string and the value of the expires property to the required value

Cookies in JavaScript: Ultimate Guide is a useful guide that describes the relevance of cookies and how to create, access and remove cookies in JavaScript.

Installing Wireshark on Ubuntu Artful

Fri, 01 Dec 2017 00:00:00 GMT

Introduction

Wireshark is a network protocol analyzer which allows inspecting network traffic at different levels. It supports all the well known network protocols and is commonly used for troubleshooting network related problems. In this article I will describe my experience with installing Wireshark and using it for finding problems with web applications

Installing Wireshark

Wireshark may be installed from source code or from a package management repository. Wireshark installation is described on the Wireshark download page.

Installation from package management repository is the simplest option but does not allow the same level of customization as installing from source code. Installing from package management repository on Debian and Ubuntu requires running the commands:

sudo apt-get update
sudo apt-get install wireshark-gtk

To install from source code, we need to first download the Wireshark source code. We have two options for installing Wireshark from source code. Either we can compile the code and generate a binary file or we can generate a package for the current operating system. The latter option is more suitable for installing Wireshark on several computers with identical operating system.

Compiling the source code

Wireshark supports two graphical interfaces. One is based on GTK+, while the other is based on QT. To generate the Wireshark binary based on GTK+, we need to run the following command:

sudo apt-get install pkg-config
sudo apt-get install gtk+-3
apt-get install libgtk-3-dev
sudo apt-get install libpcap-dev
./configure --with-gtk=yes
make
make install

After running the make command, the wireshark-gtk binary file will be generated in the current directory. The make install command will copy the binary file to /usr/local/bin/ (on Ubuntu Artful). After that we should be able to run the Wireshark using the command wireshark-gtk.

Running the command may give the following error:

wireshark: error while loading shared libraries: libwiretap.so.1: cannot open shared object

The error can be fixed by entering the command:

ldconfig

After that we should be able to run the Wireshark from command line using wireshark-gtk command.

Wireshark also has a command line client called tshark which is installed along with the wireshark-gtk binary. We can give various command line options to the configure command for customizing our Wireshark installation. For example the —with-qt option allows generating Wireshark with a QT based frontend.

Generating package for the operating system

To generate a package for the current operating system for example Debian, we need to run the command:

dpkg-buildpackage -us -uc -rfakeroot

The command requires several dependencies which may be installed using the command:

sudo apt-get install qtbase5-dev qtbase5-dev-tools qttools5-dev qttools5-dev-tools qtmultimedia5-dev libqt5svg5-dev flex python-ply \
libc-ares-dev xsltproc docbook-xsl docbook-xml libxml2-utils libcap2-dev libcap-dev bison quilt libparse-yapp-perl portaudio19-dev \
libkrb5-dev liblua5.2-dev libsmi2-dev libgeoip-dev libnl-genl-3-dev libnl-route-3-dev asciidoc cmake w3m libsbc-dev libnghttp2-dev\
libssh-gcrypt-dev liblz4-dev libsnappy-dev libspandsp-dev

Troubleshooting application problems using Wireshark and TCPDump

The problem I had was that my PHP script was making requests to Amazon Product API. The API was throttling the requests at random intervals and giving the error message that my application is sending requests too quickly.

Amazon Product API allows maximum of one request to be sent per second. My application was sending one request every 4 seconds so there should not be a problem.

I posted the error to the Amazon API forums. A similar error was reported by other users. Amazon support staff asked me to check my application using a low level tool. I decided to use TCPDump for capturing network packets while my application was running.

I ran the command:

tcpdump -w tdump.pcap -i venet0 dst 52.94.216.120

for capturing the packets. The packets were captured from interface venet0 and saved to tdump.pcap file. Only packets sent to the IP 52.94.216.120 were captured, which is the IP address of the Amazon Product API. The address appears to change every few minutes so I had to ping the API host ecs.amazonaws.de immediately before running the command in order to get the IP address.

After capturing the packets, I downloaded the tdump.pcap file from the server and opened it using Wireshark. The Wireshark displayed details of each captured packet. The Wireshark showed that the TLS Hello Client package which is sent at the start of the TLS handshake was being sent with an interval of 4 seconds. This showed that my application was sending requests to the API server with sufficient pause. I posted the tdump.pcap file to the Amazon API forums.

Conclusion

Wireshark is a useful tool for troubleshooting network related problems. It supports hundreds of network protocols and has excellent features for analyzing and presenting network protocol information.

Running multiple versions of mailx command

Fri, 24 Nov 2017 00:00:00 GMT

Certain commands in Linux can be provided by multiple packages. The command provided by each package may differ. For example the mailx command is provided by the bsd-mailx package as well as the heirloom-mailx package. The mailx command provided by the packages use different command line parameters.

To force the system to use a command from a specific package we can use the update-alternatives command. For example the command:

update-alternatives --set mailx /usr/bin/heirloom-mailx

will set the mailx provided by heirloom-mailx as the default mailx command. So when we enter mailx on the command line, it will run the mailx provided by heirloom-mailx package.

The above command may give an error on Debian Stretch. If the error suggests that the mailx package has not been registered, then the error can be corrected by registering the heirloom-mailx package using the command:

update-alternatives --install /usr/bin/mailx mailx /usr/bin/heirloom-mailx 10

See the discussion on How to change mailx?, for more information on how to run multiple versions of mailx.

Resizing Linux root partitions without unmount on Google Cloud

Thu, 23 Nov 2017 00:00:00 GMT

Virtual machines often run out of disk space and require resizing. Resizing root partitions without unmounting is always risky. It is recommended to take a backup of the partition’s useful data before resizing the partition.

To resize the root partition we need to first increase the disk size in the virtual machine configuration. Next we need to enter the command:

sudo fdisk /dev/{root_partition_name}

This will open the root partition in the fdisk program.

The next step is to print the partition table using the command ’p’. Note the starting block number of the root partition. Next we need to delete the root partition using the command ’d’. Next we need to create the root partition using the command ’n’. We will need to enter the starting block number that we noted earlier. The size of the new partition should be set to the new size. Next we need to write the partition table using the ’w’ command. After that we need to reboot the virtual machine.

See the article How to resize ext4 root partition live without umount on Linux for more information on how to resize Linux root partitions without unmount on Google Cloud

Configuring Apache mod status

Wed, 22 Nov 2017 00:00:00 GMT

Apache Mod Status is an Apache module that provides useful information about the requests being sent to Apache.

It provides information such as the PID of Apache processes, the number of active and idle threads in each process, the thread id and request handled for each thread

Apache mod_status is available by default in most Apache servers installations. To enable mod_status, we need to enable the mod_status Apache module. We then need to enable the mod_status handle within a location directive. This directive can be placed within a virtual host block. For example:

<Location /server-status>
SetHandler server-status

Order Deny,Allow
Deny from all
Allow from .example.com
</Location>

This will allow the Apache status information to be accessed on the url: http://localhost/server-status. If we add refresh=N to the URL, then the status page will refresh every N seconds. For example: http://localhost/server-status?refresh=5. The ExtendedStatus Apache directive allows extra information to be displayed with mod_status. This directive is used side wide and cannot be applied on a virtual host basis. It can slow down the server, so it should only be used if needed

Creating custom themes for October CMS

Wed, 22 Nov 2017 00:00:00 GMT

Introduction

October CMS is a content management system based on Laravel PHP Framework. It allows developing websites and web applications. It supports custom themes and plugins. In this article I will describe my experience with developing a custom theme for the October CMS.

Installation

To install the October CMS, we need to first ensure that our server meets the minimum requirements. October CMS requires PHP 7 and higher and the following PHP extensions: PDO, CURL, OpenSSL, Mbstring, Zip and the PHP GD library.

The first steps in the installation process is to download the installation zip file to the web root directory. The zip file should be unzipped and write permissions should be given to the extracted files and folders. Next the install.php script should be opened in the browser. This will start the installation wizard. The wizard will ask for the database information as well as the admin user name and password. The wizard will then download the main files. Once the wizard ends, the installation files should be removed from the server.

October CMS supports task scheduling. For this it requires adding the following entry to the crontab file:

* * * * * php /path/to/artisan schedule:run >> /dev/null 2>&1

Features

The main features of the October CMS are described on the October CMS website. October CMS is built on top of the Laravel PHP framework.

It supports the Twig template engine which is used to render web pages. It supports building pages using page components. A page component can be an item such as list of most recent posts. The October CMS can be extended using plugins. It has an easy to use administrative interface and supports easy creation of back-end pages for plugins.

The October CMS provides a simple Ajax framework. It also provides the Builder plugin, which allows easily creating fully functional plugins. The October CMS uses file based templates. It also includes a built in asset combiner which allows combining CSS and JavaScript files.

The October CMS provides a marketplace for third party plugins and themes. Another useful features of the October CMS is that it allows plugin and core updates to be installed easily. It also has a translate plugin that allows easy translation of content

Developing custom themes

Developing a custom theme for October CMS is quite easy. The CMS provides a built in theme called demo. A good starting point for developing the custom theme is to copy the demo theme folder and change the name of the folder to the new theme name.

The theme folder contains the following subfolders:

assets. This folder contains the theme assets such as CCS files, JavaScript files, Images and Fonts.
content. This folder contains the textual content of the website pages
layouts. This folder contains the layout files for the theme. A layout file is a file with .htm extension. It contains the basic HTML page and placeholders for pages and partials. The top of the layout file has a configuration section that can contain information such as the description of the layout. The configuration section is read by the CMS and ends with ’==’ on a new line.
pages. This folder contains the page files. A page file has a .htm extension. It also starts with a configuration section which can contain information such as the page title, URL and layout. The page file is rendered in the given layout and is applied to the layout automatically when the given URL is accessed by the user.
partials. This folder contains partial files. A partial file is a file ending with .htm that can be reused in other layouts, pages or partials. Examples of partial files are header and footer files that are common to several pages.
readme.md and theme.yaml. The readme.md file contains information about the theme in markdown format. The theme.yaml file contains theme information that is used by the CMS. Both files should exist in the theme root directory

The templates files inside the layout, page and partial folders can contain upto three sections, which are configuration, PHP and Twig sections. Each section should end with ‘==’ on a new line.

Each theme can be managed from the October CMS back-end. The back-end allows editing the template files in a HTML editor as well as placing the theme in maintenance mode.

Conclusion

The October CMS is a feature rich CMS that is easy to use and extend. It is suitable for developing websites and web applications

Disabling output buffering with mod_fcgid

Wed, 22 Nov 2017 00:00:00 GMT

Output buffering in PHP allows output from a PHP script to be stored in a buffer. When the buffer is flushed, the output is displayed to the user. Sometimes we may want to disable output buffering, so we can quickly access the output from a PHP script. For example it is useful to be able to view the output from long running scripts.

The output from a PHP script can be buffered by other tools such as the web server, intermediate proxy servers, web server modules, web browser, PHP Interpreter etc.

Disabling PHP output buffering is simple when using mod_php. We only need to disable the output buffering in the php.ini configuration, but when using PHP in a CGI or FastCGI environment it can be a bit tricky. Apache can use different modules for communicating with PHP in a FastCGI environment. For example mod_fcgid

To disable output buffering in a FastCGI environment, the following steps can be taken:

output buffering can be disabled for PHP by adding these lines to

php.ini:

zlib.output_compression = 0;
output_buffering = 0;

The mod_fcgid buffering can be disabled by adding: FcgidOutputBufferSize 0 to the virtual host configuration.
The buffering by Apache can be disabled by disabling the mod_deflate Apache module.

The question PHP output buffer not flushing on Stackoverflow provides useful information on PHP output buffering problems.

Exo Platform - Intranet portal and collaboration

Wed, 22 Nov 2017 00:00:00 GMT

Introduction

Exo Platform is an Intranet portal with collaboration features written in Java language. It is available as open source. It allows employees to collaborate effectively and productively. Its main features are an activity stream, documents, wiki, calendar, forums, contacts and spaces.

Installation

Exo Platform installation is well documented in this Installation and Startup guide. The Exo Platform is installed using a Docker image. Developers can install the Exo Platform from source code using the Exo Platform GitHub repository.

Installing as a Docker image

The Installing eXo Platform Community Edition guide describes how to install Exo Platform community edition from a Docker image. The Exo Platform page on DockerHub is a useful resource for installing Exo Platform community edition.

To install the Exo Platform from Docker image, we need to run the command:

docker run -d -p 8080:8080 --name=exo exoplatform/exo-community:latest

This will download the latest community edition of Exo Platform and start it on port 8080.

By default the Exo Platform uses HSQL database. It is recommended to use MySQL database for production environments. The database connection information, port numbers and other parameters can be set by passing environment variables to the Docker container. For example the command:

docker run -d -p 8080:8080 --name=exo -e EXO_ADDONS_LIST="exo-tasks:1.0.0,exo-answers" exoplatform/exo-community:latest

will install the task and answers addons and start the Exo Platform on port 8080. The different customization options are described on the Exo Platform page on DockerHub.

Installing from source

To install the Exo Platform from source we need to clone the Exo Platform source code repository. In order to install from source, we need to have Apache Maven installed.

Apache Maven is a build tool for Java. Once we have installed the Apache Maven, we need to change to the cloned source code repository. Next we need to run the command:

mvn install

This will start the build process which will create the required executable scripts. The build process installs the Exo Platform as a Tom Cat application.

To start the Exo Platform we need to run the command:

plf-community-tomcat-standalone/target/platform-community-{CURRENT_VERSION}/platform-community-{CURRENT_VERSION}/start_eXo.(sh|bat)

This will start the Exo Platform on the default port 8080 with HSQL as the database

The platform can be configured by editing the configuration files in the conf folder located in the same folder as the executable scripts. For example the file server.xml contains configuration information for the HSQL database.

To use MySQL instead of HSQL, we need to replace the HSQL configuration lines in server.xml with the contents of server-mysql.xml. This is described in the database configuration guide.

When we start the Exo Platform for the first time, we should use the command: start_eXo.sh. This will run the application in the foreground allowing us to see if there are any errors. The application can be stopped using the command: stop_eXo.sh. To run the application in the background we can use the command: start_eXo.sh -b.

Addons

The addon command is used to manage the Exo Platform addons. To list the available addons, we need to run the command: ./addon list from the same folder as the start_eXo.sh command.

This will list all the addons that can be installed on the currently installed version of the Exo Platform. An addon can be installed using the command:

./addon install exo-tasks:1.2.4

This will install the addon exo-task version 1.2.4. To install addons in a Docker based installation, we need to pass environment variables to the docker run command.

The Exo Platform has some useful addons such as exo-tasks, exo-chat and exo-onlyoffice. These addons are still under development for the current v5 of the Exo Platform. Stable version of these addons are available for v4.4 of the Exo Platform.

The exo-task addon adds project and task features to the Exo Platform. The exo-chat addon adds text messaging and document sharing features. The video-calls addon adds video chat features to the exo-chat addon allowing groups to video chat. The exo-chat addon integrates with the calendar, task and wiki features allowing tasks and events to be posted to the calendar and wiki.

The exo-onlyoffice addon allows office documents such as Power Point Presentations, Word Documents, SpreadSheets etc to be opened within Exo Platform. The addon basically opens the documents using the API of an Only Office server. The Only Office server needs to be installed before using this addon.

Features

Exo Platform has two main sections. One is the company intranet and the other are spaces. Spaces are groups that may be created for special purposes. For example one space may be created for each department in a company.

Both the company intranet and spaces support activity streams, documents, wiki, calendar, forums and contacts. The task addon can be used to add task and project capabilities to both the company intranet and spaces.

The Exo Platform also allows creating multiple sites. Each site can belong to a specific product or company department. All features of the Exo Platform can be customized and configured from the administration area. The administration area also allows managing gadgets and portlets.

Gadgets and Portlets are small applications that can be added to pages or the Dashboard. Before a Gadget or Portlet can be used, it has to be added to one or more categories. This can be done by going to: Administration —> Application.

Conclusion

The Exo Platform is a comprehensive feature rich application for developing Intranet portals. It has excellent documentation and is being actively developed

Installing Docker on Debian Jessie

Wed, 22 Nov 2017 00:00:00 GMT

Docker is a solution for managing light weight containers. To install Docker on Debian Jessie, we need to first install packages to allow apt to use a repository over HTTPS. We can do this using the command:

sudo apt-get install \
     apt-transport-https \
     ca-certificates \
     curl \
     gnupg2 \
     software-properties-common

Next we need to add Docker’s official GPG key using the command:

curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg | sudo apt-key add -

After that we need to install the apt repository using the command:

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID") \
   $(lsb_release -cs) \
   stable"

Next we need to install Docker Community Edition using the command:

sudo apt-get install docker-ce

Next we can test the installation by running the hello world Docker Container. We can do this using the command:

sudo docker run hello-world

See the official Docker installation guide for Debian for details on installing Docker on Debian Jessie

Permanently authenticating with remote git repositories

Wed, 22 Nov 2017 00:00:00 GMT

Some times we need to non-interactively run scripts which commit to git source code repositories that are hosted remotely. For example we may want push changes to a git repository stored on Bit Bucket. Normally we will be asked to enter our Bit Bucket credentials, but this can be automated. There are two ways of doing this:

We can use credential caching. This will cache our credentials for a given duration. For example use the following command to cache your git credentials for 1 hour:

git config --global credential.helper 'cache --timeout 3600'

Another option is to place a .netrc file in your home directory. This file is used by curl which is used by git when working with remote repositories. The format of this file is as follows:

machine bitbucket.org
login your-bitbucket-username
password your-bitbucket-password

The above methods should work with any remote git hosting. The article: Permanently authenticating with Git repositories describes how to permanently authenticate with git repositories.

Using Redir command for redirecting ports

Wed, 22 Nov 2017 00:00:00 GMT

Redir is a very useful command line script that allows port redirection. The script listens on the given TCP port and forwards incoming connection to the given IP address and port.

One use case for the Redir command is to setup SSH access for a container that is running inside a host system. For example if you are running LXC containers on a server and you want to be able to SSH to a container from outside the server, then you can use the redir command to redirect incoming connections on SSH port 22 or some other port to the SSH port of your container.

The following redir command listens on all available IP addresses and port 1000. It redirects all incoming connections to port 22 on host 192.168.1.6. The ’&’ at the end of the command allows the command to run in the background.

redir --laddr=0.0.0.0  --lport=1000 --caddr=192.168.1.6 --cport=22 &

Installing Zurmo CRM and Oxwell Social Network Platform

Tue, 21 Nov 2017 00:00:00 GMT

Introduction

Zurmo CRM is a customer relationship management system with a wide range of features such as Contact Management, Activity Management, Deal Tracking and more. Oxwell social network platform allows creation of social networks and online communities. Both applications are written in PHP and MySQL. Both platforms are open source and free to use.

Installing Zurmo CRM

Zurmo CRM is based on the Yii framework. It also uses JQuery and RedBeansPhp ORM library. It requires Php 5.3 and above and also requires a few extensions. The installation is described in the Zurmo installation guide. To install the Zurmo CRM, we need to first download the source code to the webroot and then extract the zip file. We then need to run the installer from the browser. This will start the installation wizard for Zurmo CRM.

The installation wizard will ask for database connection information and the superadmin user credentials.

The installation of the Oxwell Social Network is similar to that of the Zurmo CRM. The installation is described in this manual installation guide. We need to first download the source code and move it to the web root. Next we need to extract the source code and configure the virtual host for the new website.

Next we need to run the installation wizard from the web browser. The wizard will ask us for the database information and the admin user name and password.

Zurmo CRM features

Zurmo CRM features are described in this feature list on the Zurmo CRM website. The features include Contact Management, Activity Management, Deal Tracking, Gamification, Reporting, Workflow, Marketing Automation, Product Management, Internationalization and more. The Zurmo CRM has an online demo which showcases all the features.

The Oxwell Social Network has many useful features for developing online social networks. The features are documented in this feature list on the Oxwell website.

The core features include: Messages, Friends, Facebook Connect, Privacy, Contact us, Image slideshow and more. Other features include Facebook-like newsfeed, Photo sharing, Forums, Video embeds, User blogs, Groups, Social media sharing, Birthdays, Events, Chat and more.

Oxwell Social Network supports several monetization options such as User credits, Membership levels, Banner rotation, PayPal integration and more.

Administrative and Management features of the Oxwell Social Network include the ability to add new features with plugins, 1-click updates, Theme customization, Drag’n’drop CMS, Mobile CMS, Public/private site modes, Site translation, Statistics, User roles, Pre-moderate content, Appoint moderators, Mass-mail users, Code injection and more.

Conclusion

Oxwell Social Network Platform and Zurmo CRM are excellent applications for building social networks and managing customers. Both applications are feature rich and open source.

Installing Canvas learning management system

Wed, 15 Nov 2017 00:00:00 GMT

Introduction

Canvas LMS is a modern Learning Management System. It allows teachers and instructors to publish learning material online such as courses, quizzes, assignments, announcements, syllabus, modules, discussions, files, learning outcomes and more. Canvas is available as open source software and can be used free of charge.

Installation

Canvas LMS is based on Ruby on Rails framework. It uses PostgreSQL server for the database. Its installation is a bit tricky and is well described in this installation guide for production environments. The main installation steps are as follows:

Database installation and configuration

This involves installing PostgreSQL server and creating a database and database user that will be used to access the database. A system account with same name as the database user should be created and given superuser rights.

Downloading the code

This involves downloading the source code of Canvas LMS using GIT or downloading the tarball or zip file. Once the code has been downloaded it needs to be placed in the correct directory.

Dependency installation

This involves installing the software required to run Canvas LMS. Canvas requires atleast Ruby v2.4 and NodeJS v6. It also requires PostgreSQL v9.3 and above. Most of Canvas dependencies are Ruby Gems. Certain NodeJs packages are also required. Ruby Gems are software packages for the Ruby language, which can be managed by the Gems package management system.

Canvas uses the Bundler gem for managing package dependencies. Bundler may be regarded as a versioned package management system that sits on top of Gems. Bundler can be installed by entering the following command from the root of the Canvas installation:
```
sudo gem install bundler --version 1.13.6
```
The packages required by Canvas can then be installed using the command:
```
bundle install --path vendor/bundle
```
Canvas also requires certain NodeJs packages which can be installed using the Yarn package for NodeJs. Canvas prefers Yarn instead of the commonly used NPM package management system. Yarn can be installed using the commands:
```
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install --allow-downgrades yarn=0.27.5-1
```
The NodeJs dependencies can then be installed using the command: yarn install
Canvas Configuration

This steps involves copying the default configuration files to the config directory and then editing the main configuration files. The configuration files can be copied over using the following command from the installation root:
```
for config in amazon_s3 database \
  delayed_jobs domain file_store outgoing_mail security external_migration; \
  do cp config/$config.yml.example config/$config.yml; done
```
The next step is to edit the most important configuration files. These are the database.yml, domain.yml, security.yml and outgoing_mail.yml files.

Generate Assets

Once the configuration files have been copied and edited, the next step is the generate the website assets, which are basically optimized CSS and JavaScript files. The assets can be generated using the command:

cd {canvas_root_folder}
mkdir -p log tmp/pids public/assets app/stylesheets/brandable_css_brands
touch app/stylesheets/_brandable_variables_defaults_autogenerated.scss
touch Gemfile.lock
sudo chown -R canvasuser config/environment.rb log tmp public/assets \
                              app/stylesheets/_brandable_variables_defaults_autogenerated.scss \
                              app/stylesheets/brandable_css_brands Gemfile.lock config.ru
yarn install
RAILS_ENV=production bundle exec rake canvas:compile_assets
sudo chown -R canvasuser public/dist/brandable_css

Database Population

This step involves creating the required database tables and table indexes. It requires entering the command: RAILS_ENV=production bundle exec rake db:initial_setup. This will create the database tables, and will also prompt the user for the admin user name and password.
File permissions and Apache configuration

This requires creating the system user that will own the Canvas source code files. It is usually set to the user that the Apache web server runs as. For example www-data. Once the user has been created, the file ownership needs to be changed. The next steps involves installing the Apache Passenger module.

This is used to execute the Ruby source code. Once Passenger module has been installed, we need to create a virtual host for serving the Canvas files. After that our installation is completed and we should be able to access our Canvas installation at the URL specified in the Apache virtual host configuration file. The installation guide should be checked for the exact steps required for configuring Apache.

Features

Canvas has all the features needed for creating virtual learning environments. All learning material is contained within courses. Each course can contain learning material such as quizzes, assignments, syllabus, announcements, grades, modules, student progress and more.

The structure of Canvas is hierarchical. It consists of accounts and sub accounts. Each account and sub accounts contain courses. Each course contains quizzes, modules etc. The Canvas user guide describes all the features of Canvas.

Canvas also has a messaging feature which allows users to send messages to other Canvas users. Other features include mobiles apps, calendar for managing events and a dashboard that displays overview of each course. Canvas also has user and group management

Conclusion

Canvas is a very comprehensive and feature rich learning management system. It has been under development since 2008 and is actively supported by the open source community.

Data mining with Graylog2

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

In my earlier post titled, Graylog2 - A scalable open source centralized logging system, I had described my experience with installing and configuring Graylog2. In this post, I will describe my experience with using the search features of Graylog2.

Using Graylog2 for analyzing log files

Once you have installed and configured Graylog2, you will of course want to use it for analyzing your log messages. The Graylog2 web interface has several features that can be used to analyze log files. These features include:

Input

Inputs allow us to receive log messages. Different types of inputs can be defined using the Graylog2 web interface. Graylog2 supports many input formats. In short you can send log information from almost any device or application to your Graylog2 server. This information can be analysed and viewed using the Graylog2 web interface.

The most commonly used inputs are Syslog TCP and Syslog UDP. If a Syslog server is running on your Graylog2 server then you can use these inputs. The Syslog server can work as a remote Syslog server for other hosts on your network. So the hosts send their log files to the Syslog server running on the Graylog2 server which in turn forwards them to Graylog2.

Syslog is a well known service for sending and receiving log messages. Syslog clients and servers are available for most operating systems. Most Linux based operating systems have build in Syslog clients.

To configure a Syslog client to send log messages to a Syslog server all you need to do is define the IP address of a Syslog server. For example on Linux distributions running the rsyslog client you only need to add this line at the end of the rsyslog.conf file: . @@ip_of_syslog_server:514. The Syslog server should be running on your Graylog2 server. Thats all you need to do in order to send Syslog messages from your hosts to your Graylog2 server.
Search

This feature allows you to search your log messages directly. To access this feature click on the Search link at the top. Then enter your search terms in the search box. This will return all Graylog2 messages that contain the search terms.

Even if your Graylog2 server does not have much RAM or CPU and has tens of millions of messages, your query should not take more than 2 seconds. We can also save a search and can go back to a previous search with a single click. The Graylog2 documentation has good description on the search features. For example the query: source: server_name, will return all messages sent by a server called server_name.
Streams and Alerts

A stream is a flow of messages that satisfies certain conditions. For example all messages that contain the term SSH can be placed in one stream. Several such rules can be applied to a Stream. Alerts can also be added to the Stream. An alert is triggered if a certain condition is fulfilled.

For example an alert can be triggered if more than a certain number of messages enter a stream in a certain period of time. We can also configure responses to alerts. For example if an alert is triggered in a Stream, an email can be sent to configured users or a HTTP post notification can be sent to a web script.

It is also possible to define outputs for a Stream. A Stream of messages can be sent out of a configured output. e.g to a certain destination IP address and port. The Graylog2 documentation has useful information about Graylog2 alerts and Graylog2 streams.
Dashboards and Widgets

Dashboards and Widgets provide actual statistical information about your messages. We can create several Dashboards. Each Dashboard can contain several Widgets. A Widget shows statistics from a search. For example when we do a search, the results can be saved as a widget.

It is also possible to upload Content Packs using the Graylog2 web interface. A content pack is a set of predefined Streams, Widgets and Dashboards. For example we can upload a content pack for the Nginx web server that provides us with a dashboard. The Graylog2 documentation has useful information about Graylog2 Dashboards.
Extractors

The real benefit of searching messages is when we know the structure of the message and can define fields for each message type. Once we have defined field types we can run precise search queries on the fields.

Extractors are a feature of Graylog2 that allow us to do this. An Extractor is basically a regular expression that should match a field in a message. For each input we can define several Extractors and can perform search queries using fields defined by Extractors. Extractors can be imported and exported allowing us to share Extractors with the Graylog2 community.

Conclusion

Graylog2 is an excellent logging system with many features. I have installed and configured Graylog2 in a minimum setup. Graylog2 can also be configured in a large setup that supports tens of thousands of logging devices. Such a configuration involves load balancers and message brokers but is similar in principal to the small setup.

Graylog2 provides very good community support. If you need help with some features you can always ask on the Graylog2 forum. You can also post bugs to the Graylog2 issue tracker

Graylog2 - A scalable open source centralized logging system

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

In this blog post I will describe my experience with installing an open source centralized logging system called Graylog2. Graylog2 is an excellent log management system. It is based on well known open source tools such as Mongodb and Elasticsearch. It allows I.T administrators to collect logging information from almost any source, including Linux and Windows computers.

Graylog2 works by collecting log messages from different sources. The log messages are saved in a central database. This database is queried and the query results are displayed on a nice web interface. Alerts can be configured that notify the administrator in case certain terms appear in search logs.

Installing Graylog2

Graylog2 installation is very simple. It requires a Linux server with Java. Graylog2 provides package repositories for Debian, Ubuntu and Centos based Linux distributions.

Installing Graylog2 is as simple as installing the package repository and then entering the command:

yum install graylog2-server graylog2-web

on Centos and

apt-get install graylog2-server graylog2-web

on Debian and Ubuntu. After installation a few fields need to be changed in the Graylog2 configuration files and thats all!. The installation is described in the Graylog2 documentation.

The standard Graylog2 installation involves 2 main components graylog2-server and graylog2-web. Large scale deployments of Graylog2 are more complex and involve a component called Graylog2-radio.

Conclusion

Graylog2 is a scalable log management system that can handle Terabytes of data. It is a useful tool for collecting and analyzing log data.

High CPU problems with Pfsense

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

Recently our Pfsense gateway server was consuming too much CPU. Especially during high network activity. This blog post describes my experience with trying to optimize Pfsense.

Pfsense is a network gateway system. High load on the network gateway can result in all sorts of issues for the client user. e.g slow download speed, problems with VPN connectivity, slow remote desktop connections etc. Hence tuning the network gateway for performance is very important.

Pfsense is based on the FreeBSD operating system, which is similar to Linux. Pfsense has an excellent support forum with thousands of topics related to Pfsense. The support community is large and helpful and you are likely to receive a reply within hours.

High CPU problem

All the posts related to High CPU use in Pfsense mentioned following factors. Ipv6 traffic, Device polling and Tuning kernel parameters.

I read that tuning kernel parameters is a bad idea unless you are using Pfsense for a very specific purpose. I followed the topic “Something is causing High CPU load” on the Pfsense forums and blocked IPV6 traffic from the System->Advanced->Networking->Allow IPv6 option. That did not lower the CPU usage.

I then read the articles “FreeBSD Set Network Polling To Boost Performance” and “Polling and FreeBSD”. The articles suggested that on FreeBSD systems with heavy hardware utilization, disabling device polling can reduce CPU usage.

The reason was that normally hardware devices interrupt the CPU each time they have data that needs to be processed. This interruption of the CPU creates an overhead which increases with the number of interrupts.

So if the network traffic is high then the network card will interrupt the CPU and will cause high CPU usage.

Solution: Enable device polling and Virtio drivers

Device Polling

The solution is to enable device polling which disables CPU interrupts. With device polling, the hardware device, e.g network card is polled by the CPU at regular intervals. This reduces the CPU overhead. Enabling device polling is very simple in Pfsense. You just have to disable the “Disable device polling” option under System->Advanced->Networking. This actually helped us a lot. The CPU usage went down from 80% to 40% and the load went down from 4.0 to 3.0.

Virtio Drivers

This was still a bit high. Our Pfsense server was running as a virtual machine under Proxmox virtualization platform. Proxmox supports virtio drivers for hard disks and network cards. Virtio drivers allow virtual machines to access the host servers hardware directly and provide performance close to that of physical hardware.

Unfortunately Pfsense does not support Virtio drivers by default. The Pfsense wiki article, “VirtIO Driver Support”, has a useful guide on enabling support for Virtio drivers. We replaced all the virtual network cards with VirtIO based network cards on our Pfsense virtual machine. We then loaded the FreeBSD VirtIO kernel modules as described in the wiki. After enabling support for VirtIO network drivers, the CPU usage became normal and we also got a good increase in network performance.

The instructions for enabling VirtIO drivers are very simple. You have to first load the VirtIO kernel modules by editing the /boot/loader.conf.local file. Then you have to disable TCP offload features from the Pfsense gui. You can do this by selecting the checkbox options related to TCP offloading.

TCP Offloading

TCP Offloading is a feature provided by some operating systems that transfer TCP/IP processing to the network interface card. Its especially useful on Gigabit and Multigigabit network interface cards.

Transferring TCP/IP processing has the obvious advantage of speeding up network traffic. But it does have its disadvantages. For e.g network cards do not have advanced resource scheduling capabilities and security mechanisms of operating systems.

TCP Offloading is supported by Windows and FreeBSD operating systems. Linux kernel does not support TCP Offloading out of the box and requires special support.

Pfsense allows offloading certain TCP/IP stack functions such as checksum calculations and TCP segmentation. These functions have to be disabled in order to get the VirtIO drivers to work under Pfsense. Hopefully future releases of Pfsense will have built in support for virtio network cards with support for TCP offloading.

Conclusion

What actually helped us the most was enabling support for VirtIO network cards. For now our Pfsense server is working well and stable. Tuning your important servers such as Network Gateways is important and can lead to many long term benefits.

Installing Atlassian products on Google Cloud

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

Atlassian is a software company that develops products for software developers and project managers. Its main products are an Issue Management system called Jira and a Document Management and Collaboration platform called Confluence.

Other products by Atlassian include a group chat system called Hip Chat, An Identity Management System called Crowd, A Version Control System for Source Code called Bit Bucket and a Continuous Integration platform called Bamboo.

In this article I will describe my experience with installing, configuring and integrating Jira, Confluence, Crowd and Hip Chat. We used the cloud version of Hip Chat. The other products were self hosted and installed on a separate LXC container on a Google Cloud instance. Each product was accessed through a Nginx proxy server.

Atlassian provides two versions of each product. A cloud version which is hosted by Atlassian and a server version which can be hosted by us. Atlassian products are written in pure Java and are run from a Tomcat container.

Multiple Atlassian products cannot be run from a single Tomcat container. All products require the latest installation of Java Development Kit. (JDK). Atlassian products require significant hardware resources.

Installation of the products is well documented and easy to follow. The Atlassian support staff is helpful and its forum is informative. All Atlassian products have a minimum fee of 10 USD per year. According to Atlassian, this amount is donated entirely to Charity. All Atlassian products support addons. Addons can be used to integrate Atlassian products with each other and with third party applications.

Jira Issue Management

Jira is an issue management platform. It can be used for issue tracking, bug tracking and project management. Jira is offered in three packages. Jira Core, which is the base software. Jira Software, which is intended for use by software development teams and Jira Service Desk, which is intended for I.T support and Help Desk staff. All three Jira packages may be used together. Jira functionality can be extended with the help of addons.

Installing Jira

Jira requires about 2 Gb RAM. Installation of Jira is very simple and is described in this Jira Installation Guide. To install Jira, we need to first download the installer. The installer is run from the command line and it prompts for a few questions such as the installation directory, TCP ports etc.

Once the installer has run, Jira should be accessible from a web browser on port 8080. We then have to run the setup wizard. The setup wizard has an automated install option and a manual install option. If we select the manual install option, then we are first asked to select a database type and enter our database credentials.

Next we will be asked to enter the general site information such as site name, base URL for the site etc. After that we need to enter our license. Then we need to enter the admin account information. After that we need to enter the mail server information. The Jira should now be ready to use.

Jira Features

The main features of Jira are project management and issue tracking. A project contains issues. Issues can be viewed on a Kanban board or as a list. Issues can have custom fields and can be searched using different criteria. Issues may also be updated in bulk. Jira provides dashboards which give a summary of the issues. Jira users can be managed locally or from a directory server such as Atlassian Crowd. Crowd supports user management and Single Sign On (SSO) for Jira.

Capture for Jira is an addon for Jira that allows creating Jira issues from a Google Chrome extension. For example a software developer who is working on fixing a bug on a web page, can create an Issue in Jira using the Jira Chrome extension. The extension displays a screenshot of the web page that the developer is working on. The developer may annotate this screenshot. The developer can then create a new Jira issue with the screenshot as an attachment

Confluence Document Management

Confluence is a document management system. It provides spaces within which documents are managed. A space is a grouping of functions related to document management. For example within a space, we can create wiki pages, write blog posts or post questions and answers.

Installing Confluence

Confluence requires about 6 Gb RAM. Installation of Confluence is very simple and is described in this Confluence Installation Guide. To install Confluence, we need to first download the installer.

The installer is run from the command line and it prompts for a few questions such as the installation directory, TCP ports etc. Once the installer has run, Confluence should be accessible from a web browser on port 8090. We then have to run the setup wizard. In the first step, we will be asked to select the installation type. We can choose either production or evaluation. The production option requires an external database, while in the evaluation option the database is included with the installation. In the next step we will be asked to enter our license.

In the next step we will be asked to select a database type and enter our database credentials. Next we need to choose if we want to populate our site with example data. Next we need to choose how we want to manage our users. The available options are managing the users from Confluence or managing them from Jira. Next we need to enter the admin account information. Next we need to enter the mail server information. The Confluence should now be ready to use.

Confluence Features

Confluence allows us to manage documents, write blog posts, post questions and answers and more. All Confluence features are grouped into spaces. Confluence also allows importing content from other Atlassian products using Application Links. Application links is an addon that allows Atlassian products to import content from other Atlassian products. For example we can display Jira issues on a Confluence page using a Jira Issue Macro. This requires that both the Jira and Confluence are connected with Application Links

Crowd Identity Management

Crowd is an identity management platform. It provides centralized user management and Single Sign on for all Atlassian products. It also provides an OpenId server called CrowdId, which allows integrating third party applications using the OpenId protocol.

Installing Crowd

Crowd requires about 256 Mb RAM. Installation of Crowd is very simple and is described in this Crowd Installation Guide. To install Crowd, we need to first download the archive file. Next we need to specify the location of the Crowd home directory. This can be done by editing the file: {CROWD_INSTALL}\crowd-webapp\WEB-INF\classes\crowd-init.properties. The next step is to connect Crowd to an external database. Next we need to run the startup script start_crowd.sh in the Crowd installation directory. This will start Crowd on port 8095

Crowd Features

Crowd allow us to manage our users from a central location. These users can be authenticated by different applications. Each application must be registered in Crowd. Users in Crowd are stored in directories. Crowd supports the following types of directories:

Internal. An Internal directory stores authentication and authorization information in the Crowd database
Delegated authentication. Delegated authentication directories store the users and groups in Crowd, but delegate the authentication to an external LDAP server
Connector. A connector directory is a LDAP connector that stores the user and group information in an external LDAP server
Remote Crowd. A remote Crowd directory uses another Crowd directory stored on a remote Crowd instance
Custom. A custom directory allows developers to implement their own interfaces for connecting custom data stores to Crowd

Hip Chat

Atlassian Hip Chat allows chatting, video conferencing and screen sharing for teams and businesses. Hip Chat supports rooms where software developers can discuss work. All Atlassian products link with Hip Chat. For example the Jira Hip Chat integration allows developers to add chat rooms to a Jira issue. When the issue is updated, a notification is sent to the chat room on Hip Chat. Confluence and Bit Bucket have similar integrations for Hip Chat. For example each time a Confluence page is updated, a notification can be sent to a room. Similarly, when a commit is made to a repository in Bit Bucket, a notification can be sent to a chat room in Hip Chat.

Conclusion

Atlassian products are suitable for the enterprise and software development teams. The products are very well integrated with each other and have very useful features.

Installing HumHub and Communote on Debian Jessie

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

HumHub and Communote are both web based social network platforms that allow teams and small organizations to communicate and collaborate using Activity Streams, Messaging, Blogging, Tagging and more. HumHub is based on the Yii Php Framework, while Communote is written in Java.

Installing Communote

Communote installation is well documented in this installation guide on the communote website. There are two main options for installing Communote on Linux based servers. We can either install Communote from an installation package or from a WAR file.

Both installation options require a database server. Oracle, Microsoft SQL server, PostGres and MySQL database servers are supported.

Both installation options require Java Runtime Environment version 8 and the JDBC driver for the selected Database management system. Installation from WAR file also requires Apache Tomcat server. Both installation options can be completed in a few steps as described in the installation guide.

After the Communote has been downloaded and installed, it can accessed from the browser at http://localhost:8080. The startup page shows a web based wizard that allows the Communote to be configured.

Installing HumHub

HumHub is written in PHP and like most Php based applications, it is easy to install. The installation is described in this installation guide. To install HumHub, we first need to create a new MySQL database. Next we need to download the HumHub files and configure the Apache virtual host for serving the files. The installation can then be completed from the web based installation wizard.

Communote Features

Communote features and installation are well documented on their support website. Communote is essentially a Micro Blogging platform. It allows users to enter notes which can be viewed in an activity stream. Users can comment on notes and also search within notes. Notes are categorized by topics and tags.

Communote has a client server architecture. The Communote service installed on the server can be accessed from the browser as well as mobile and desktop clients. Communote can be integrated into other applications using embed codes and widgets. The embed code allows Communote activity streams to be embedded in any website, while the widgets allow the a simplified view of Communote to be embedded in other websites. Communote also allows notes to be submitted by email.

Communote functionality can be extended by plugins. Plugins such as language, message queue and single sign on are supported.

HumHub features

The main feature of HumHub is spaces. A space is a collection of features such as files, users and groups. The features available in a space are determined by the modules that have been activated for the space.

For example modules such as wiki, tasks, polls, sms can be enabled for individual spaces. Each space contains an activity stream by default. A user dashboard shows all activity that has occurred within spaces that the user is subscribed to. The layout of the HumHub is responsive and works on mobile devices and desktop browsers.

Conclusion

HumHum and Communote are both useful platforms for collaborating and forming social networks. Both platforms are open source and free to use.

Installing NextCloud, Collabora and Spreed on Debian Jessie

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

NextCloud is a document management and collaboration system. It allows users to access their images, documents, spreadsheets and other files from any location and any device.

NextCloud is free and open source and is a fork of OwnCloud. It also has an enterprise edition.

NextCloud consists of a server and clients. It can also be extended by apps. NextCloud server has a web interface that allows users to change the server settings and also manage their files. The web interface allows access to all the features of NextCloud. NextCloud clients are desktop and mobile applications that allow access to documents.

Installation

NextCloud requires about of 512Mb RAM. It supports most modern Linux based operating systems. It requires Php 5.6+. It supports modern database servers including MySQL/MariaDb, PostGres and Oracle. It also supports all major web browsers, desktop and mobile clients.

NextCloud can be installed either from source or from SNAP. Source installation is slightly more difficult. It requires installation of several Php extensions. To install NextCloud from source, we need to first download the NextCloud files.

After that we need to create a database and then run the NextCloud installer from a web browser. This will install the NextCloud server. The details of the installation and the different installation options are well documented on the NextCloud installation guide.

Apps

NextCloud functionality can be extended with Apps. An App is a plugin that adds a new feature to NextCloud. NextCloud Apps are available from the NextCloud App Store. An App can be installed and enabled from the Apps page.

It is also possible to develop a custom App and upload it to the NextCloud installation folder. NextCloud provides Apps for creating notes, video conferencing, editing office documents, managing tasks, creating diagrams, file sharing, file versioning and much more.

Collabora Online

Collabora Online is a LibreOffice based online document editor. It supports all major office document formats including Microsoft and Libre Office documents. Collabora Online is free and open source and works in all modern browsers. It supports collaborative editing and supports integration with other applications such as NextCloud.

Collabora Online server can be installed easily from a Docker image. The following line of code will start a Docker container running Collabora server. It will download the Docker image of Collabora, if not already downloaded. It will start a container running Collabora server listening on port 9980.

docker run -t -d -p 9980:9980 -e "domain=example.com" --cap-add MKNOD collabora/code

The installation of Collabora Online server with Nginx reverse proxy is described in the article Running Online Office with Nextcloud and nginx. The Collabora Online server must be accessible over the internet using HTTPS protocol.

Self signed certificates may not work so a HTTPS certificate should be obtained from a Certificate Authority. A good option for obtaining HTTPS certificates is Lets Encrypt. It provides HTTPS certificates free of cost.

The Collabora plugin for NextCloud is easy to configure. It requires the public url of the Collabora server. Once the plugin has been configured, we can start editing office documents from within NextCloud using Collabora Online.

Spreed Video Conferencing

Spreed is a video conferencing server. It implements the WebRTC protocol, which is supported by most modern web browsers. Spreed supports audio and video chat and conferencing. It also supports screen sharing and text based chat.

Spreed can be installed as a Docker container using the command:

docker run --rm --name my-spreed-webrtc -p 8080:8080 -p 8443:8443 -v `pwd`:/srv/extra -i -t spreed/webrtc

Installation of Spreed is described on the Docker repository page for Spreed.

Once the Spreed server is running, it will listen on port 8443. Like the Collabora Online server it also requires HTTPS. Next we need to enable and configure the Spreed plugin for NextCloud. Configuration involves specifying the public URL of the Spreed server.

Conclusion

NextCloud is a useful tool for managing documents online. It is easy to install and is well supported by an active online community. It is included in this list of the Best Cloud Storage tools of 2017.

Open Project - Collaborative Project Management

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

Open Project is an online project management application with collaboration features. It is one of the most popular open source project management software.

Open Project is written in Ruby and Angular Js. It is similar to Redmine but has more features out of the box. For example features such as Agile & Scrum and Mobile support are not present in Redmine by default but may be added using plugins. Also Redmine has better support for managing software projects, while Open Project supports all types of projects.

Features

Open Project provides support for all stages in a project’s life-cycle. Starting from Project conception and initiation to Project definition and planning, Project launch, Project performance and control and finally the Project closing stage.

Features provided by Open Project include: Gannt Chart, Calendar, Roadmap, Issue Tracking, Activity Streams, Meetings, Agile & Scrum, Cost Reports, Budgets, Wiki, News and Forum. Open Project functionality can be enhanced using plugins. Open Project has an enterprise edition with more features than the community edition. It also has support for mobile devices.

Installation

Open Project installation is very simple and is described in the Open Project Installation Guide. To install Open Project on Debian Jessie we need to first install the packager.io repository signing key using the command:

wget -qO- https://dl.packager.io/srv/opf/openproject-ce/key | sudo apt-key add -

Next we need to install the HTTPS support for apt using the command:

apt-get install apt-transport-https

After that we need to add the OpenProject package source to the package manager source list. This can be done using the command:

sudo wget -O /etc/apt/sources.list.d/openproject-ce.list https://dl.packager.io/srv/opf/openproject-ce/stable/7/installer/debian/8.repo

After that we can install the OpenProject software using the commands:

apt-get update
apt-get install openproject

This should install Open Project to the folder: /opt/openproject.

Configuration

Once the installation has ended, we need to configure the Open Project using the command:

openproject configure

This will start the configuration wizard.

The configuration wizard will prompt the user for a number of questions such as the MySQL server information, Git/Sub version integration options, Email configuration, HTTPS certificate settings, Web server configuration options and more. The configuration wizard also allows us to enter a path prefix for the openproject installation URL.

Before starting the configuration wizard, we should have the MySQL database connection information, email settings and location of the HTTPS certificate files. If we don’t have this information with us, then we can cancel the wizard at any stage and then run it again. The wizard will resume from where it was interrupted. If we want to start the configuration process again we can issue the command: openproject reconfigure.

When the configuration process ends, it starts the openproject server on port 6000. It also configures all the required virtual host settings for Apache, if this option was selecting during the configuration. We should now be able to login to our Open Project installation at the URL that was specified during the configuration process.

The default login information is admin/admin. Once the user enters this information, he will be asked to enter a new admin password

Conclusion

Open Project is a useful project management tool. It supports all stages of a Project’s life-cycle. Its design is user friendly with support for mobile devices. Its a good choice for managing projects

Problems with accessing websites over local network using HaProxy and Pfsense

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

Recently I had problems with accessing websites over a network that was using HaProxy and Pfsense. The websites could be accessed without problems over the internet, but it was not possible to access the websites over local network. For example when connected to the network over VPN

The Network Topology

Our network uses Pfsense as the gateway server. It uses HaProxy load balancer package for serving websites. The HaProxy allows us to run several websites spread over different virtual machines over a single IP address.

Problem with accessing websites

We could easily access the websites over the internet, but if we tried to access the website over the local network using the actual host name of the website, we got connection time out errors. It was simply not possible to access the website over local network.

After days of troubleshooting the problem turned out to be very simple. The HaProxy package of Pfsense has an option called “Use Client-IP to connect to back end servers”. This option forwards connections to the back end web servers so it seems to the web servers that the connection request was sent directly by the client. But this option also adds some firewall rules that cause problems with accessing the website directly. As soon as we checked off this option we were able to access the websites over local network without problems.

Conclusion

The HaProxy package for Pfsense has some great features that greatly simplify hosting multiple websites. It has support for HTTPS and also supports load balancing and fail-over.

Pydio - Document management and Sharing

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

Pydio is a Secure File Management and Sync solution. It was formerly called AjaXplorer. Pydio stands for “Put Your Data In Orbit”. It allows users to manage and share their files across devices. It has a customizable web interface and has clients for Desktop and Mobile devices.

Features

Notable features of Pydio include support for different storage systems such as FTP, NFS and Dropbox. Support for multiple user management directories such as LDAP, Active Directory, Radius etc. Pydio is Open Source and provides a REST API.

Its features can be extended using plugins. It provides a flexible dashboard for viewing server statistics and document usage. Pydio also supports Collaborative editing using Collabora Online.

Other notable features of Pydio include “Federated Cloud Sharing”, which allows users to share their data by connecting their Pydio clouds. Pydio also provides a “File Gateway”, which allows users to serve public links over a dedicated URL.

Installation

Pydio may be installed from package repository or from source. Installation from source is easy and is described in the article Install from archive. To install from source, we need to first download the source code to the installation folder. After that we need to create a database and then run the installer from the web browser.

Conclusion

Pydio is a useful document management and sharing solution. It is similar to NextCloud but does not have as many plugins. Pydio can provide a useful frontend for storage systems. Here is a list of the top free Cloud Storage providers of 2017, that can be used as storage systems for Pydio.

Securing your Email server

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

In this blog post I will describe my experience with installing, configuring and securing an email server. Many companies have their own email servers but one thing I noticed about in house email servers is that they are very unreliable.

A very high percentage of emails sent to custom email addresses like user@abccompany.com cannot be delivered. The most likely cause is careless configuration of the email server. Email is a critical part of the I.T infrastructure so implementing email correctly is very important.

The email server I installed and configured is called Postfix. Postfix is one of the most popular email servers. It runs on most Unix based servers such as AIX, BSD, HP-UX, Linux, MacOS X, Solaris etc. Best of all Postfix is free and open source.

Installing and configuring Postfix Email Server

Installing Postfix email server requires entering the command: yum install postfix on Centos servers. After that you have to edit the main.cf and master.cf configuration files. Postfix website has some useful documentation on basic configuration of Postfix.

Some of the important Postfix configuration parameters are: myorigin, mydestination, mynetworks, relaydomains, relayhost, myhostname, mydomain and interfaces. These parameters need to be changed in /etc/main.cf. After the parameters have been updated, reload Postfix configuration with this command: postfix reload. You should now be able to send email through your Postfix server from any device.

Installation and basic configuration of Postfix will result in a fully operational mail server. But it does not guarantee reliable error free mail delivery. To ensure that your email is delivered reliably your mail server needs to be secured.

Securing your email server

Email relies heavily on DNS. Most of the techniques for securing email require changes to your domains DNS settings. Some of the steps that can be taken to secure email servers are:

Open Relay prevention

Open Relay prevention should be the first step for securing your mail server. An Open Relay is an email server that allows anyone to send email to anyone. Open Relays are often used by malware for sending spam. To prevent Postfix from working as an open relay configure the relay_domains, mynetworks_style, mynetworks and relayhost parameters in main.cf and reload Postfix. Use an online tool such as Open Relay Test for checking if your mail server is working as an open relay.

Spam prevention

Postfix has built in protection for preventing spam. The smtpd_recipient_restrictions and smtpd_sender_restrictions parameters in main.cf file can be used to reduce spam.

Sender Policy Framework

SPF is a security mechanism that restricts the mail servers that are allowed to send email for a given domain. It involves adding a TXT DNS record for your domain.

This record specifies the IP addresses of the servers that are allowed to send email for your domain. When a mail server receives an email from your domain, it will check if your domain has a SPF DNS record. If is has a SPF record, then it will check if the server that sent the email is allowed to do so by your SPF DNS record.
Reverse DNS

DNS maps host names to IP addresses, while Reverse DNS does the opposite. i.e it maps IP addresses to host names. When an email server receives an email it may do a reverse lookup on the IP address of the server that sent the email. If the host name corresponding to the IP address does not match the host name of the sending email server then the email may be rejected.

DomainKeys Identified Mail

DKIM is a security mechanism that involves creating an encrypted email signature and verifying the signature. Like SPF, DKIM also relies on the DNS. To enable DKIM, the user has to first install a DKIM server. e.g on Centos that can be done with the command: yum install opendkim. The main steps involved in configuring Open DKIM are:
- Generate a public/private key using the opendkim-gen command.
- Specify the path of the keys in the opendkim.conf configuration file.
- Give the name of the DKIM server in Postfix main.cf configuration file.
- Enter the generated public key in a new TXT DNS record for your domain.
- After that the email messages you send through your mail server will contain a DKIM header field which contains the signature of the message. The receiving email server retrieves the private key from the DNS and uses it to verify the message. If the check fails, then the message may be discarded.
Domain Based Message Authentication, Reporting and Conformance (DMARC)

DMARC is a system for validating emails. It requires use of SPF and DKIM. It is the latest and most popular email validation system. It not only ensures that the received email is from a valid sender but also ensures the email has not been tampered with. It requires publishing encryption information in DNS records. DMARC allows the administrator to specify policies for handling email including policies on reporting the email.

According to the Wikipedia entry for DMARC, in October 2013, Mailman added support for DMARC and set the default DMARC policy to p=reject. In 2014 Yahoo and AOL both set their default DMARC policy to p=reject.

According to the article “Google to adopt strictest DMARC policy to fight spam, phishing”, Google will transition to the policy p=reject in June 2016. The policy p=reject implies that email that cannot be validated by DMARC will be rejected. In March 2015 DMARC RFC 7489 was published.

Conclusion

Secure DNS and Email settings greatly increases the reliability of your email server. They reduce the chance of your emails ending up as spam. There are some excellent on-line tools that can help you find problems with your Email and DNS configuration.

For example dnsstuff, mxtoolbox, emailsecuritygrader, nmonitoring, centralops.

Having your own secure Email server can be very useful. You can customize it to your needs and integrate it with other services. You can achieve this goal with open source tools such as Postfix.

Setting up your own Content Delivery Network using Bind DNS

Tue, 14 Nov 2017 00:00:00 GMT

Introduction

Recently I worked on constructing a simple Content Delivery Network using Bind DNS. I will share my experience in this blog post. Bind DNS is an open source DNS server. It is the most widely used DNS server. DNS is one of the building blocks of the internet. The main use of DNS is to translate domains names to IP Addresses. DNS also has other uses such as securing email.

Main function of a Content Delivery Network (CDN)

The main purpose of a Content Delivery Network is to deliver content such as images, audio/video files, JavaScript and CSS files.

The CDN is configured to deliver content with low latency and high availability. Most of the data on social networks and popular websites such as WordPress and E-commerce websites is delivered using a CDN.

Anycast DNS used by CDNs

Although CDNs use different protocols and technologies for delivering content securely and with high performance, a common feature of most CDNs is their use of Anycast method for resolving DNS queries.

Anycast is a network addressing and routing methodology by which data is sent to the nearest node. When Anycast is used with DNS, The DNS server returns the IP address of the host that is closest to the client that sent the DNS query. The client then fetches the content from this host.

Split DNS

Split DNS is a technique supported by the DNS protocol which allows a DNS server to return different information depending on the source of the request.

Bind DNS implements Split DNS with a feature called Views. To configure Views on a Bind DNS server you have to first create one View statement for each Geographical zone that needs to be served.

Implementing Split DNS using Bind Views

Views are given in the Bind configuration file, named.conf. The article “DNS BIND view Clause” has useful information on DNS Views. Then you have to add DNS records to each view. The records are returned to the client that has access to the View.

Client access for views can be configured by adding match-clients clause to each View. Match-clients clause allows us to specify the IP addresses that have access to the View. These addresses can be given using Bind Access Control Lists or simple network address notation such as 192.168.1.0/24.

Views are a very powerful feature of Bind DNS that allows DNS records to be served depending on the client. Usually the DNS records are served depending on the client’s geographical location. Configuring Views in Bind DNS is very simple.

Problems with using Bind Views

A problem with using Bind Views is that if the client users are spread across the world, then you will need to enter the network addresses of all possible clients. This can make your Bind configuration quite large.

For example if you have one CDN server in Country A and one in Country B, then your Bind configuration will need to have all possible network addresses of Country A and Country B. So you will need one view statement for Country A and one view statement for Country B.

Each view statement will need to have a match-clause for that country. Clients located in Country A will then match view for Country A and will be served DNS records in that view. The IP addresses in these records should be of servers located in Country A.

Same for Country B. What if a user is not in Country A or Country B. We will need a default View for such clients. This view can be defined using the clause: match-clients{ any;}.

Views in Bind are processed in the order in which they are defined, so the default view should be defined at the end. The main difficulty in implementing Split DNS using views is that you need to know the IP address range for countries. You can use a free database from MaxMind. However you will need to convert the network addresses in the database to a format that is compatible with Bind ACLs.

Fortunately there are scripts available that allow us to do this. The article “HOWTO Implement GeoDNS using BIND” demonstrates a script that downloads MaxMind database and converts network addresses to a format compatible with Bind ACLs. The script provides a file with all the converted network addresses grouped by country. Once you have configured the Bind views, all you need to do is upload content to your CDN servers. Your users should then be able to download files from the closest server.

Conclusion

Bind is a very powerful DNS server that has many features. Some of the different configurations of Bind are described in DNS Configuration Types.

DNS implemented using Bind forms the backbone of many important services such as email security, web browsing, content delivery networks etc. A good understanding of Bind will make it easier to learn about the different services that use DNS.

Pros and Cons of Technology in the light of Religion

Thu, 05 Oct 2017 00:00:00 GMT

Introduction

Modern technology has transformed our lives. It has brought tremendous changes to culture, society and moral and social values. In this article I will try to express my views on how technology is affecting our lives and how religion can guide us in understanding technology.

Challenges of Internet and mobile

Undoubtedly, the internet has created many new opportunities for people. Interacting with people over the internet has its own costs. It requires concentration and effort on the part of the individual.

Understanding how to work efficiently over the internet is something that takes time and effort. These days interaction over the internet or mobile takes a significant part of a person’s time and energy. Since the internet is not a natural medium of interaction between people, spending excessive time on the internet can lead to physical and social problems.

I think an unfortunate fact of today’s world is that the internet and mobile have diverted people away from natural physical interactions. The internet and mobile are not part of man’s natural environment. Today they may seem in-disposable, but a few decades from now they could be regarded as outdated. We should not let technology overwhelm us. Technology is at our disposal. It is upto us if we use or misuse it.

We may think that technology is breaking up our societies. However I disagree with this statement. It is the misuse of technology that has created this impression. The misuse of technology has created paranoia, social and health related problems. Understanding how we are shooting ourselves in the foot with technology requires peaceful reflection on our own existence and why the Almighty created us and sent us in this world. This is where religion can help us.

Technology in the light of religion

Technology has undoubtedly created social and health problems. The solution to this problem I believe is an understanding of religion. A person who understands and practices religion not only helps him self but also others.

Religion is present in some form or the other in every society. It may seem outdated and old fashioned but it is in fact the solution to the most modern and sophisticated problems affecting individuals and society.

The outcome of our actions reflects our belief system. We are judged by what we do as individuals and as a society. If we do good, we are appreciated for it. If we do bad we are condemned for it. Our success is measured by the good deeds that we do. No matter how much or how little science and technology we develop, whether we live in skyscrapers or in caves, we will be judged by our good deeds. Understanding this simple fact requires an effort in understanding and practicing religion.

Not all people are gifted with the same amount of intellect or abilities. We have very little control or knowledge of the world around us. We should consult with the Almighty who created everything that we see and that which we cannot see. The Almighty did not just create the world and let it run on its own. He is very much alive and present and is waiting for us to ask him for help.

Whatever peace is left in the world is because of the Almighty’s generosity and mercy. We should be grateful to the Almighty for all the blessings he gave us, is giving us and will give us as long as we live.

True religion I believe does not prohibit or condemn technology. Nor does it impose barriers on our lives. I believe religion is here to guide us and help us to overcome our problems. It is up to us to impose the correct barriers and restrictions on our lives in the way that benefits us.

Everything outside our bodies, including our clothes, is not a part of us. The world around us poses a challenge to our brains and bodies. Technology is just another challenge for us.

This does not mean we should blame science and technology for all the problems of the world. Many of today’s problems appear to be caused by misuse of science and technology. For example weapons of mass destruction are highly sophisticated products of science and technology.

The problem I believe is not science and technology itself but rather it is intention with which it is being used which is the problem. Why are we using our own brains and capabilities to destroy our own selves?. The answer to this problem I believe lies in reflecting upon the universe around us.

We should start with nature since it has been and will always be present, independently of our own existence. We should reflect upon how the wonderful ecosystem of nature works and how much dependent we are on it. In short we should seek guidance from books like the Holy Quran which help us to reflect upon the reality of our own existence and help us to achieve the peace that we so greatly need.

Understanding science and technology in the light of Islam

Islam I believe is the only true religion. It is the most complete and comprehensive religious guide. It is equally applicable to the most simple and the most sophisticated of societies. Islam has been provided to us by a most merciful being who knows all and sees all. It is the system we should accept and use for improving our lives and our societies.

How does Islam view science and technology?. Well in Islam science and technology are termed as Knowledge. Islam provides specific and clear guidelines on how to use this knowledge so we and our societies benefit from it.

The application of Islam is upto us and our societies. We are free to practice Islam as we wish. It does not matter where we live or what our languages are. The message of Islam is universal and applies to all irrespective of their language or race.

In order to understand Islam, we need to be sound in mind and body. There are five pillars of Islam. These are: Belief in one God, Establishment of Worship, Fasting, Charity and Pilgrimage. Understanding of Islam requires practice and patience. Once we start with basics of Islam, our understanding and faith will grow and we will notice a positive change in our selves and our lives.

Conclusion

In this article I have tried to discuss how Technology is affecting our lives and what role religion can play towards improving our understanding of technology and the world around us.

Adding drag and drop to list items

Mon, 02 Oct 2017 00:00:00 GMT

In order to drag and drop HTML list items, we need to set the draggable attribute of the list items to true. We also need to define event handlers on the list items for the following events: dragstart, drop and dragover.

The dragstart event is called when the user starts to drag a list item. In this event, the user should do three things. Firstly set the allowed drag and drop event to ‘move’. Secondly save the inner html of the list item and thirdly save the list item object or an id of the list item object.

The drop event is called when the user drops a list item. The event handler for this event should first prevent the default drop action of the browser, which is to open the dragged item in a new tab. The saved inner html of the original list item should be fetched and set to the innerHTML property of the current list item. The original list item should be fetched and its innerHTML property should be set to the innerHTML property of the current list item.

The dragover event is called when the user drags a list item. The event handler for this event should prevent the default dragover action of the browser, which is to open the dragged item in a new tab.

The following code can be used:

/** Listen for dragstart events */
list_item.addEventListener('dragstart', function(ev) {
  /** The allowed event is set to move */
  ev.dataTransfer.effectAllowed          = 'move';
  /** The html content of the list item is saved */
  ev.dataTransfer.setData('text/html', ev.target.innerHTML);
  /** The current list item object is set */
  current_list_item           = ev.target;
}, false);

/** Listen for drop events */
list_item.addEventListener('drop', function(ev) {
  /** The default drop action is prevented */
  ev.preventDefault();
  /** The saved list item html data is fetched */
  var data                               = ev.dataTransfer.getData("text/html");
  /** The current list item html */
  var list_item_html                     = ev.target.innerHTML;
  /** The list item html is set */
  ev.target.innerHTML                    = data;
  /** The original list item html is set */
  current_list_item.innerHTML = list_item_html;
}, false);

/** Listen for drop events */
list_item.addEventListener('dragover', function(ev) {
  ev.preventDefault();
}, false);

The above code allows list items to be dragged and dropped. When a list item is dropped, the dropped item and the item on which the drop occurred are swapped.

See the W3school’s guide on Html 5 Drag and Drop for more information on HTML 5 Drag and Drop.

New features in HTML 5

Fri, 14 Apr 2017 00:00:00 GMT

HTML 5 is the latest version of the HTML specification after HTML 4.01. The HTML standard is maintained by the W3C.

HTML 5 contains many new features. The basic structure of a HTML 5 document is as follows:

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
</head>
<body>
</body>
</html>

HTML 5 has very loose type checking and does not require closing ’/’ for void tags such as meta tag or img tag. It also does not require enclosing quotation marks and optional attributes such as type attribute for script and link tags.

Several tags are now considered as obsolete in HTML 5. e.g frame and frameset tags.

Some new structural elements introduced in HTML 5 are: section, hroup, article, nav, aside, header, footer, address.

HTML 5 supports following inline elements: (bold text), (italics text), (for emphasizing text).

HTML 5 also provides support for accessibility. The WAI-ARIA standard makes web content and web applications more accessible to people with disabilities. One of the main features of WAI-ARIA is landmark roles. The role attribute can be added to any HTML element. The value of the role attribute indicates the use of the HTML element. e.g <nav role=“navigation”> implies the nav tag is used for navigation. The role attribute can have fixed values.

HTML 5 provides the video and audio elements. It allows playback of video and audio files. The video element is used just like the img element. A video element can include several nested source elements. Each source element is a video source file. The audio element works similarly.

HTML 5 also allows offline access. To allow a HTML 5 application to be accessed offline, we need to add the manifest attribute to the HTML tag. The value of this attribute is the URL of the manifest file. The manifest file lists the resources that should be available to the application when it is offline. For example the images, CSS and JavaScript files

New features in CSS 3

Fri, 14 Apr 2017 00:00:00 GMT

CSS 3 is well supported in all major browsers. Old versions of Internet Explorer such as Explorer 6 support most CSS 3 modules. Modules such as Font Face module are not supported in old browsers.

Rounded Corners

In CSS 2.1 rounded corner affect could only be achieved using background images. CSS 3 provides the border-radius declaration which produces rounded corners. e.g .round {border-radius: 10px}

Vendor Prefixes

Vendor prefixes are strings that are prepended to CSS 3 properties that are not supported by all browsers. e.g -khtml- prefix is for Konqueror web browser. -rim- prefix is for Rim browser. -ms- prefix is for Internet Explorer browser. -o- prefix is for Opera browser. -moz- prefix is for Mozilla Firefox browser. -webkit- prefix is for Chrome and Safari browsers. CSS styles are cascaded which means the styles lower down the order are applied later. Also more specific CSS selectors take precedence over less specific CSS selectors.

Multiple Columns

CSS 3 also allows dividing content into columns. For example it can divide content into columns of certain width or certain number of columns. The following code shows how this works:

<div id="main">
  column 1 text
  column 2 text
  column 3 text
</div>
<style>

#main {
  column-width: 12em; /** It will set the width of each column equal to 12 em; * OR use below style: */
  column-count : 3; /** It will divide the content into 4 columns */
}
</style>

Word Wrap

CSS 3 provides the word-wrap style which allows wrapping a word if it exceeds the bounds of its container. e.g #main {word-wrap: break-word;}.

Attribute Selectors

CSS 3 provides attribute selectors. It allows selecting a HTML tag with a certain attribute. e.g:

img[alt] {
  border: 3px dashed #e15f5f;
}

Applies a dashed border to all images on the page with alt attribute.

img[alt="film"] {
  border: 3px dashed #e15f5f;
}

Applies a dashed border to all images on the page with alt attribute that starts with “film”. If alt*=“film” was used instead, then it would match all images with alt tag containing the text film. If alt$=“film” was used then it would match all images with alt tag that ends with film. CSS 3 also allows id values to start with a number. e.g id=“1940year”.

CSS3 structural pseudo-classes

CSS 3 also introduces the following selectors known as pseudo-classes: :nth-child(n), :nth-lastchild(n), :nth-of-type(n), and :nth-last-of-type(n), where n can be a number, arithmetic progression, odd, even. e.g:

nav ul li:nth-child(2n+3) a {
 color: #fe0208;
}

will apply the color to all list items with position 3, 5, 7, 9 and so on.

nav ul li:not(.internal) a {
 color: #fe0208;
}

will apply the color to all list items with class not equal to internal and that contain <a> tag.

li:: first-line {color: red;} will apply the color red to the fist line in the list tag.

p::after {
    content: " - Remember this";
}

will apply the content after the paragraph tag

Custom Web Typograhpy

CSS 3 also allows the use of custom font types using the @font-face syntax. e.g:

@font-face {
 font-family: 'BebasNeueRegular';
 src: url('BebasNeue-webfont.eot');
}

Will load the font file BebasNeue-webfont.eot and make give it the font-family name of BebasNeueRegular. Following are the main types of fonts used for websites: Embedded OpenType (EOT), TrueType (TTF), Scalable Vector Graphics (SVG) and Web Open Font Format (WOFF).

Responsive font size unit

CSS 3 introduced the vw unit, which implies percentage of the viewport. e.g: h1 { font-size: 8vw }, implies the font size is 8% of the viewport width. e.g if viewport width is 200mm, then font-size of h1 elements will be 16mm.

New Color formats and Alpha transparency

CSS 3 introduced following new color formats: RGB, HSL. RGBA and HSLA allow specifying the alpha transparency. Alpha transparency allows the opacity value of a certain tag to be set. For example a certain portion of a page can have a different opacity then the other portions.

Some useful websites:

Server Monitoring with Munin

Fri, 14 Apr 2017 00:00:00 GMT

Introduction

Server monitoring is a useful task that is performed by server administrators. Server monitoring involves keeping a check on the various parts of a server such as its CPU, Memory, Hard Disk, Network Traffic, Services and more. Server monitoring can avoid problems that can disrupt services. For example a timely alert for a failing hard disk can allow the server administrator to replace the hard drive before its too late.

Munin is a well know server monitoring tool that allows monitoring all aspects of a server. Although it is regarded as a network monitoring tool, it can do much more than monitor network traffic. It can infact monitor most services. These services may be on switches, routers or servers. Munin uses Round Robin Database (RRD) tool for storing the data.

Munin works by collecting data from the services being monitored. This data is stored in RRD files. The data in these files is then displayed on graphs. Round Robin Database (RRD) tool is a standard for storing data which varies with time such as network bandwidth, CPU load, temperature etc.

Architecture

The Munin architecture consists of three main parts. The Munin master, the Munin Node and Munin plugins. The Munin master and nodes have a client server relationship. In a typical Munin installation, there is one Munin master and one or more Munin nodes. The Munin nodes may collect data from remote devices such as switches, routers, firewalls, gateways etc using standard protocols such as SNMP and SSH.

The job of the Munin master is to poll the Munin nodes at regular intervals for data. It stores the data in RRD files. It sends alerts in case the data values exceed certain thresholds. For example it can send an email if the CPU usage exceeds a certain value.

The Munin master also generates graphs from the data. It consists of following components: munin-cron is the main script that runs the Munin master. It is run every 5 minutes by default. It calls the scripts munin-update which collects data from Munin nodes and munin-limit which checks if the data values are within the threshold limits.

The Munin node runs on one or more servers as a cron job. It listens on TCP port 4949. It executes the Munin plugins and returns the data for each plugin to the Munin server.

Munin Plugins are basically scripts that return the time series data for the service being monitored as well as information on how to graph the data. Munin plugins can be written in any programming language, although they are usually written in Perl.

Installation

Munin can be installed either from source code or from package repositories. The Munin installation guide strongly recommends installing from package repositories.

On Centos, Munin is part of the default package repositories and can be installed with the command:

yum install munin munin-node

This will install the latest stable version of Munin. As of now the stable version of Munin is 2.0.30. The development version of Munin is 2.999.6. The development version has a much better user interface as can be seen from this demo.

Configuration

After installing Munin, the next step is to configure it so it monitors the required services. To configure Munin we need to add the host names of the Munin nodes in the Munin server configuration file. This is usually /etc/munin/munin.conf.

We also need to copy the plugin files to the Munin service directory which is /etc/munin/plugins. The service directory contains symbolic links to the actual plugins files. There are two types of Munin plugins. The core plugins which are part of the Munin installation and the Third party plugins which are developed by various contributors.

The core plugins are part of the Munin installation and are usually located in /usr/share/munin/plugins. To Install a Munin plugin, we simply have to create a symbolic link inside the service directory which points to the actual plugin file. For example:

ln -s /usr/share/plugins/http_loadtime /etc/munin/plugins/

After that we need to restart the Munin node service using the command:

service munin-node restart

The munin-node service needs to be restarted after changes to the Munin configuration files or plugins. Third party plugins may be downloaded to any location. They can be installed by simply adding a symbolic link to the actual plugin file.

A Munin plugin usually monitors one service. However it is possible for a Munin plugin to monitor several services. A Munin wildcard plugin can monitor several services. By convention the name of a wildcard plugin ends with ”_”. For example the mysql_ core plugin is a wildcard plugin. It monitors several services such as MySQL network traffic, MySQL slow queries, mysql innodb buffer pool etc.

To get the list of services monitored by a wildcard plugin we can run the command:

munin-run plugin_name suggest

The munin-run command allows us to test run the plugin. To monitor one of the services supported by the mysql plugin for example the mysql slow queries, we can use the following command:

ln -s /usr/share/munin/plugins/mysql_ /etc/munin/plugins/mysql_slow

We can have several plugins that all point to the same mysql_ plugin file.

Troubleshooting

Munin errors are stored in log files located at /var/log/munin and /var/log/munin-node. The most important log files to monitor are: /var/log/munin/munin-update.log, which contains log messages reported by the Munin master during data collection and /var/log/munin-node/munin-node.log, which contains log messages reported by the Munin node.

Sometimes Munin log files report plugin timeout errors. An interesting approach to fix such problems was discussed on Server Fault. The question is titled: “Munin mysql plugins results inconsistent, with IPC::Sharelite store() error: Identifier removed”. It discussed errors reported by the Munin mysql_ wildcard plugin.

This plugin uses shared memory for storing data. Sometimes access to the shared memory can be blocked by a semaphore. The solution to the problem was to use strace tool to find the semaphore id and then use the command:

ipcrm -s semaphore-id

to remove the semaphore. This solution was very useful for me as it fixed all the errors reported by Munin node.

Munin provides a web interface which displays the graphs of the monitored services. Configuring the web interface can be a bit tricky because of SeLinux or server management tools such as Plesk or Cpanel.

The current version of Munin which is 2.0.30 contains a virtual host configuration file for Apache server which simplifies the task of configuring the web interface. This configuration file may need to be updated depending on the Munin installation paths.

The development version of Munin contains a built in web server called munin-httpd which handles the task of serving html files and graph images. The munin-httpd can easily be served over a reverse proxy server such as Nginx or Apache.

Munin graphs and HTML files can be generated in two ways. They can either be generated at regular intervals using the munin-graph and munin-html scripts or they can be generated on demand using the cgi scripts munin-cgi-graph and munin-cgi-html. Since graph generation can consume significant amount of CPU and Memory, some users may prefer to generate the graphs on demand. Configuration of on demand graph generation requires modification of the Apache virtual host configuration file and can be a bit tricky.

Conclusion

Munin can monitor almost any time related activity with the help of plugins. There are Munin plugins for monitoring different types of services such as LDAP, Snort, Bind Dns, Mysql, Apache etc.

There is even a Munin plugin for monitoring the PageSpeed score of a web page. While Munin runs only on Linux based operating systems, it can monitor services running on Windows operating system. Munin is a very useful Server Monitoring tool with many features.

What are CSS Media Queries

Fri, 14 Apr 2017 00:00:00 GMT

CSS Media Queries allow us to use CSS rules for particular media

Media Queries are a new feature of CSS3. CSS 2.1 also supported Media specific styles. For example the media attribute of the link tag allows us to use different style sheets for different media. For example print or screen media.

CSS Media Queries are used inside CSS Style Sheets. They have the format: @media query {css_rules}, where query is the CSS query and CSS_rules are the CSS rules that are applied when the query returns true.

The query parameter can have different values such as the device width. e.g the following media query sets the background color to red if the screen width is 960px or less.

@media screen and (max-width: 960px) {
 body {background-color: red;}
}


@media screen and (max-device-width: 960px) {
 body {background-color: red;}
}

The above media query sets the body background color if the maximum device width is 960px or less.

Multiple media queries can be used and separated either with and or with ’,‘. ’,’ has same affect as ‘or’ operator.

What are CSS Grid Systems

Fri, 14 Apr 2017 00:00:00 GMT

CSS Grid Systems are CSS style-sheets that contain commonly used CSS classes. These CSS classes can be applied to HTML elements such as div tags. They provided functionalities such as tables and grids. The CSS classes are responsive so they work on devices of different sizes.

Following are some commonly used CSS Grid Systems:

CSS Grid Systems can be used to create the main layout for a website. More advanced CSS Grid Systems such as the Twitter Bootstrap Library are complete responsive frameworks, that allow construction of fully responsive websites.

Advantages of HTML 5 validation

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

In this blog post I will discuss briefly the importance of HTML 5 and how to create standards compliant HTML code.

Unified Programming language

HTML 5 is the latest version of HTML that is currently supported on all major web browsers. It is widely used in the development of not only web sites but also mobile applications and even desktop applications. For example Windows 10 has built in apps that can be created using simple HTML and JavaScript. HTML 5 can provide a unified programming language that can be used to create applications that can run on a wide range of devices.

Built in support for audio and video

HTML 5 provides built in support for many of the features that previously required third party libraries. For example video and audio are supported by default in HTML 5. We no longer need to learn third party libraries in order to embed video and audio.

W3C HTML 5 Standard

W3C is an international organization whose main job is to create Web Standards such as HTML 5. The W3C HTML 5 specification describes the correct syntax of HTML 5. Code that is written according to these specifications is considered to be correct and well written.

W3C has a web based tool on http://validator.w3.org/. It allows us to see if our code conforms to W3C standards. HTML 5 code that conforms to standards will be supported by more devices and its performance is easier to predict. It will also be easier to process by automated tools such as search engines.

Conclusion

One of the main differences between HTML 5 and previous version of HTML is that HTML 5 disallows the use of tag attributes related to styling. e.g table width. These attributes need to be expressed using CSS. With a little practice it should be easy to write standards compliant HTML 5 code. The W3 Schools website is a useful resource for learning about HTML 5.

Backup your data!

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Recently I had the misfortune of losing months of precious work. This blog post is about my unlucky experience of loosing data. Hopefully it reminds the I.T people who read this article about the importance of a good backup. While working for a client I deleted some important files by mistake, and guess what there was no backup !. Well there was a backup but it was 2 months old. At least all was not lost.

The problem

It happened while I was managing a Linux based server for a client. I entered the normal rm * -rf command, but from within the wrong directory!. That got me in a panic. I searched the internet for data recovery solutions but could not find one that could be used.

I only had SSH access to the server and could not unmount the disk that had the missing data. Thankfully I had an old backup. I thought my client would be angry at me, but strangely, he also had problems with his data backup. So he was understanding and considerate about the whole issue.

Conclusion

Even the most skilled I.T workers can make mistakes. A good secure backup is essential in any I.T environment. Its easy to get distracted and loose precious data by mistake. My advice is to have a good backup before starting work!.

Closing Unnecessary email accounts

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

If you want to keep your email organized, then it might be a good idea to have just one email account. Recently I decided to organize my email by closing email accounts that I don’t use frequently. This article provides some points on how to organize your email.

Keeping track of several email accounts can create stress and reduce productivity. Accessing all your email from one account can greatly simply work routines.

I think its a good idea to close email accounts that you do not use frequently. But what if you have important email in those accounts. Or what if you cannot close your email accounts. Fortunately there is a solution for both these cases.

Transferring email from one account to another

If you want to close your email account but still keep your email, then you can transfer your email from the account you want to close to some other email account. You can do this using a tool called ImapSync. ImapSync is a useful tool that lets you transfer email from one IMAP account to another.

The tool has no graphical user interface and is run from the command line. The tool is written by Gilles Lamiral in Perl language. It supports all major operating systems. i.e Windows, Linux and Mac OS X.

Installation guides are available for all major operating systems. Once the tool has been installed it can be run using the command imapsync. Here is an example usage of the command:

./imapsync --host1 test1.lamiral.info   --user1 test1   --password1 'secret1' \
           --host2 test2.lamiral.info   --user2 test2   --password2 'secret2' \
           --automap --justfolders --dry "$@"

The above command copies all email from the host test1.lamiral.info to test2.lamiral.info. The user name for test1.lamiral.info is test1 and for test2.lamiral.info it is test2

The automap option automatically maps the source IMAP folders to the destination IMAP folders. The justfolders option only creates the folders while the dry option implies no data is actually transferred. These two options are useful for testing the email transfer without actually transferring any email

To see all the options of the imapsync tool you can use the —help option. To start the email transfer you have to remove the justfolders and dry options. This will start the transfer of emails from host 1 to host 2. In this case from test1.lamiral.info to test2.lamiral.info

Transferring of emails can be a bit slow depending on the bandwidth of your email servers. If you want to transfer several thousand emails then it can take days to complete the transfer. Fortunately the tool caches some of the data so next time you run the tool the email transfer resumes from where it stopped

Forwarding emails from one account to another

If you dont want to close your email account, then it may be possible to configure the email account so it forwards all your email to another email account. Most email providers support email forwarding. Gmail in particular has very good support for email forwarding

Conclusion

Email is a very popular means of communication. However it can become difficult to manage if its not organized. ImapSync is a very useful tool that allows you to transfer your email messages between different servers. It can be used to consolidate all your email in one account. The tool is very popular and is well documented.

Continuous integration and Project Management with Jenkins and Redmine

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Recently I had the opportunity of installing and configuring Jenkins Continuous Integration Server and Redmine. Both of these tools are becoming increasingly popular because they simplify and organize the Software Development Process

What is Jenkins

Jenkins Continuous Integration Server is a great tool that provides an interface combining all the steps involved in the Software Build Process.

It can automate many of the tasks that can take up a lot of time and attention of the software developer such as: compiling code, deploying code to different locations, running code tests, committing code to version control repositories, updating issue tracker etc

Jenkins has almost 1,000 plugins that integrate it with other systems such as Github, Bazaar, Redmine, Bitbucket etc. Jenkins is a server based system written in Java that runs in a Servlet Container such as Apache Tomcat. It can be accessed from the browser as well as the command line.

Jenkins itself does not perform tasks such testing or compiling code, it only provides a front-end for code testing and compilation tools. For example it allows us to add tools and scripts and run them from a user friendly interface.

Installing Jenkins

Jenkins runs on port 8080 by default. To run Jenkins over https, you can easily configure Apache with mod_proxy so it serves Jenkins over https.

Installation of Jenkins is very simple. On Redhat based systems you can use the command yum install jenkins to install the Jenkins server. The Jenkins wiki describes the installation steps in detail.

Running builds in Jenkins

Jenkins user interface is very easy to follow. You create projects and then configure them by specifying scripts that need to run when the project is built.

After a project is configured it can be run by simply clicking on a button. This is called running a build. A build essentially runs the scripts that are configured in a project. The result of the build is indicated by a circle. If the circle is red then the build is said to have failed. If the circle is green then the build passed. If one of the scripts in the project returns an error code then the build fails.

Use Case: Deploying WordPress with Jenkins

Deployment Scenario

I had used Jenkins to simplify WordPress deployment. I was developing WordPress themes and plugins. I had setup a development site where I did the development work. There were two copies of the live site that were load balanced. Each site was behind a varnish caching server. So every time the development site was updated, both live sites had to be updated and all varnish caches had to be cleared.

Furthermore JavaScript and CSS files had to be copied over to the Content Delivery Network Server. Also the cache files of the live sites had to be cleared as well.

Deployment Script

Manually clearing cache files and copying content from CDN took a lot of time and effort. I decided to write a script for automating the deployment. The script takes following parameters:

user name of a support user account with sudo privileges. this account should exist on all servers
host name of the live server
password for the support user
password for MySQL user on live server
password for MySQL user on dev server
user who owns the WordPress files on live server. this is usually same as the user under which Apache runs. e.g Apache or www
deployment option. this option indicates what needs to be copied to the live server. It can have following values: ‘all’ implies copy all WordPress files and database, ‘database_only’ implies copy only the database to the live servers, ‘custom_theme_and_plugin_files_only’ implies copy only the custom WordPress theme and custom WordPress plugin files. I use this option when I make some small changes to the custom plugins or theme. Its useful because it only copies the plugin and theme files.
deployment type. this can have two values. “dev_to_live” which copies the website from dev site to live sites and “live_to_dev” which copies the website from live sites to dev site.

Adding the script to Jenkins

I wanted to run this script from a user friendly interface, so I decided to add it to Jenkins. To add the script to Jenkins I followed these steps:

Created a new project and then under the project configuration I checked the option “This build is parametrized”. I added a single parameter called “deployment_option”. This parameter specifies what needs to be deployed. It corresponds to the **“deployment option” **parameter in the script. It allows the user to specify what needs to be deployed. e.g theme and plugin files or database.
Clicked on “Add build step” and then selected “Execute Shell”. After that I entered the path to the deployment script. I added two build steps, one for each live server. Each build step deploys the website to a live server.
After that I clicked on “Post build action” and then Email-Notification. Then I entered my email address. This sends me an email in case a build fails.
After that I checked “Trigger builds remotely (e.g., from scripts)”. This generated a string token. After that I clicked on save. Thats all I had to do to configure Jenkins to deploy my WordPress site.

After that each time I wanted to deploy my WordPress site, I would first click on the project name and then “Build with parameters” option. I then updated the deployment_option parameter depending on what I want to deploy and then click on build. After that Jenkins runs the deployment script.

While the script is running the status shows as a blinking dot. Once the script has run, the dot changes to a stable red or green depending on whether the deployment script returned an error or not.

Redmine

Redmine is a well known web based Project Management application, written in Ruby language. Its features include:

Issue Tracking
Flexible role based access control
Gantt Chart and Calendar
News
Documents
File Management
Web feeds and email notification
Per Project Wiki and forums
Simple Time Tracking
Custom fields, Integration with well known version control systems including [SVN](https://en.wikipedia.org/wiki/Subversion_(software), CVS, GIT, Mercurial (software), Bazaar (software) and Darcs
LDAP authentication
Several plugins that can be used to extend Redmine

Redmine has an excellent user interface and is very easy to use. I used Redmine for organizing the development of a website based on WordPress. I found the Issue Tracker, Time Tracker and Document manager features to be very user friendly with lots of features.

Installing Redmine

Installation of Redmine is not as simple as with Jenkins. The Redmine Wiki describes the installation steps in detail.

Conclusion

Both Redmine and Jenkins have plugins that allow integration with each other. Both tools are popular especially with large software development teams.

Creating your own WordPress themes and plugins

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

WordPress is a great platform for building innovative applications. You can not only build standard websites with WordPress but you can also create complex applications full of features.

WordPress has many plugins and themes that can help you meet the requirements of even the most ambitious of customers. Recently I had the opportunity of creating custom WordPress themes and plugins for a customer. I will describe my experience in this blog post.

Twitter Bootstrap for meeting initial requirements

Initially my customer had asked for a simple website that describes the products and services of the company. That was easy I thought. All I had to do was to download a simple HTML template and enter the content provided by the customer. I like the twitter Bootstrap framework. It is a responsive framework, which means it is designed to be accessible from devices with different sizes. I really like their Jumbotron example and decided to use it as the template for the website.

Using web based code editor

The customer wanted to enter the content by himself so I had to make it easy for him to edit the HTML. I installed a web based code editor called https://icecoder.net. This is an excellent tool that allows you to easily edit code straight from the web browser without installing any software on your local computer.

IceCoder is written in PHP and like most PHP applications is easy to setup. All you need to do is to download it on your server and extract the downloaded file into a folder.

Organizing the code

I created the files: header.php and footer.php. I removed the header and footer HTML code from the Bootstrap example and moved it into these two files. Now if we wanted to create a web page it had to include the header and footer files. Once I had done that, the customer could easily create new web pages.

Setting up the development and production sites

We moved the website to a dev domain and put the main website in maintenance mode. I downloaded a sample maintenance mode web page. It had an option called notify us, which was not working. I tried to connect it to MailChimp.

Placing the production site in maintenance mode

MailChimp is an excellent service for managing newsletters and email campaigns. I decided to connect their API to the maintenance mode web page. So while the customer’s website was being developed under the dev domain the live domain showed a coming soon web page.

Users who wanted to be notified by email when the website will be live could enter their email address. MailChimp would take care of notifying us and also the users.

Moving over to WordPress

Like most customers, this customer wanted more features. Like contact forms, ability to create articles etc. I knew from past experience that this was probably the tip of the iceberg. More requirements could easily come up. I needed a content management system (CMS), that had lots of plugins and was easy to use. Drupal was one option, but I did not find it easy to use. I decided to use WordPress. It turned out to be a good choice.

Migrating site to WordPress

The customers website was ready but now it had to be moved to WordPress. The best way to do this was to create a new theme for WordPress and then create new WordPress pages. After that it would be easy to copy content from old website to the new website. WordPress.org has excellent guides on creating new WordPress themes and plugins.

Like many web developers I tried to find an easier way and instead of reading about WordPress themes, I searched for a free WordPress theme based on Bootstrap. I came across a useful article called “How to build a responsive WordPress theme with Bootstrap”. It shows how to create theme based on Twitter Bootstrap for WordPress.

The tutorial was very useful. After following it, It was much easier to read the WordPress guides on themes and plugins. I got a good understanding of WordPress and felt much more confidant about meeting the customers requirements.

Meeting the customer’s requirements

The customer wanted a website like Digital Ocean. My first reaction was that this is out of my scope. However the customer was patient and sent his requirements in small bits and pieces.

Meeting those requirements with the help of WordPress plugins turned out to be quite easy. Some of the requirements could be met using plugins, But some of the requirements required creating our own plugins. For example the requirement of custom login page could be met with the Custom Login plugin.

But how could we have tutorials, questions and projects that looked like the Digital Ocean community section. We were able to meet this requirement by using WordPress custom post types.

Custom post types are a great feature of WordPress that allow us to add new types of posts. These custom types are fully integrated into WordPress and are supported by hundred of plugins. Custom post types also support hierarchies so we can have child types of a custom post type. For example if you want to create a Hotel Management System using WordPress, then you can use Custom Post Types. A Hotel can be a custom post type. A room can be a sub type of the hotel and so on.

We were able to easily create custom post types for tutorials, questions and projects with the help of the Custom Post Type UI plugin. We then needed to display them in tabs. Easy Responsive Tabs plugin was the solution.

Now we needed a way to query the tutorials, questions and projects and display them inside the tabs. I installed the display posts shortcode for displaying custom posts. With this plugin all you need to do for displaying your custom posts is to add a short code to a page. WordPress then replaces the short code with the list of articles in your custom post type.

We needed the custom posts to look stylish like on Digital Ocean. We decided to clone the plugin and modify it according to our needs. After that we were able to easily display tutorials, questions and projects inside tabs.

Conclusion

Within a few weeks, I was able to create a WordPress site that looked like Digital Ocean. Best of all I learned a lot about WordPress development. I think the main reason for meeting the customer requirements was close collaboration with the customer.

Developing a Twitter Crawler

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Recently I had the opportunity of developing a web based Twitter analytics application for a client. The application is called HQFollowBack. During the development of the application I learned how to use software development tools and apply software development techniques for solving certain problems. I will share my experience in this blog post.

I had developed HQFollowBack a few years ago. However since then Twitter upgraded their API to version 1.1 which was not backward compatible with the previous version. As a result the HQFollowBack application stopped working. My client had invested considerable time and effort in the development of the application and did not want his efforts to go to waste. We had to upgrade the HQFollowBack application so it works with the new Twitter API.

I had written the HQFollowBack application at a time when I did not have much experience with commercial software development. The application was not easy to maintain, and as a result could not be easily updated to use the new Twitter API.

I decided to rewrite the application from scratch. It proved to be a good learning experience. Although I had a few years of experience with commercial web development, this was the first time I really paid attention to software development tools and methodologies.

Software development methodologies:

During the development of the application I used the following software development methodologies and tools:

Dont repeat yourself (DRY)

This is a Software Development Methodology that requires elimination of repetitive work. I applied this methodology in the following way:

The application has one entry point, so all requests pass through one function. The function routes the requests to different objects such as the Search and Stats objects. These objects are responsible for the search and statistics related functions of the application.

The idea was to simplify the application as much as possible. Instead of having multiple controllers,models and views, the application simply uses a single routing function that dispatches the request to the correct handler function. All requests are handled in the same way by simple handler functions.

One of the handler functions displays the user interface. It simply reads a template file and replaces place holders in the file with the correct values.

The application is basically a crawler that extracts information from Twitter. In the first version of the application it used an invisible Iframe for running the crawler code. This is not an efficient approach.

The new version of the application is 100% Ajax based. So once it loads in the browser there is no need to refresh the browser. I think this is a good structure for developing crawler applications that need to run from the browser.

Dividing the work into small components

My approach to developing the application was to divide the work requirements into small sections that could be completed in 1 or 2 days. I created a simple to do list with following fields.

Issues. Problems faced during development
To Do. List of tasks that need to get done
Done. List of tasks completed
Reference. List of website links that were useful during application development

Function caching

I think function caching is a useful feature especially in applications that do a lot of data reading. For example an application that reads data from databases or from public APIs can greatly benefit from function caching.

Function caching allows the output of a function to be cached. When the function is called for the first time, it saves its output to a cache. The next time it is called it simply returns the content of its cache. I implemented function caching in the application which greatly improved its performance.

Error handling and logging

The application is based on Php. It uses the Php functions set_error_handler and set_exception_handler for registering error and exception handling functions. This allows handling of all applications errors and exceptions.

In case an error occurs, the error handling function is called which logs the error and also sends an email to the administrator. I also added try/catch code blocks to the Php and JavaScript functions. I think its important not to overuse the try/catch code blocks. They should be used only when a specific error needs to be reported. If there is no need to report a certain error, then we can use try/catch code block on the main function that includes all the other functions.

Table Data Gateway

This is a design pattern that allows an application to access data through an object. It can be considered as a special case of the Gateway Design pattern. The Gateway Design pattern allows all access to data and configuration information through a single interface or gateway.

This makes it easier to update the application configuration information and methods used to access data. My implementation of the Gateway Design Pattern allows all access to data and configuration through a single object. This object provides helper classes for carrying out different tasks needed by the application. It simplifies the application workflow and makes the code easier to maintain.

Custom libraries

I think an important aspect of Software Development is the creation of your own Software Libraries. These days its easy to grab hold of free open source code and use it in your application. While this approach can get the job done, it may make the application difficult to upgrade. Its much better to use code that you understand and have tested.

In the HQFollowBack application I created my own custom library of utility functions that I plan on reusing and improving in future projects.

Open Source Libraries

I used following free open source code for developing the HQFollowBack application:

Php library by ricardoper on Github for accessing Twitter API.
Php library by esimaken for displaying the search results in pages. Colorbox JQuery library by jacklmoore for displaying nice popup boxes.
Api by detectlanguage for detecting the language of the user Tweets.

Code comments

I commented all Php and Javascript code used by the application. I used the phpDocumentor website as a reference on how to document Php code.

Development website

I used a separate website for the development of the application. I think its a good idea to do the development work on a separate development site. The live site should always be well tested and functional.

Software development tools:

I used the following Software development tools for developing the HQFollowBack application:

Backend testing (phpunit).

Phpunit is an excellent tool for unit testing Php code. By regularly running the unit tests I can be sure the application is working. These days applications are more and more dependent on third party tools and APIs. Changes to these external tools can break your application. It is therefore very important to test your application thoroughly and if possible run the unit and function tests automatically.

Frontend unit testing (Qunit)

Qunit is an excellent tool for unit testing JavaScript code. The Qunit cookbook is a one page comprehensive guide on using Qunit. Qunit has good support for unit tests that involve ajax calls. I used Qunit for testing the ajax calls made by the HQFollowBack application.

Frontend functional testing (Selenium)

Selenium works as a browser automation tool that can be run from programming languages such as PHP or from the Firefox browser itself. Selenium allows recording website activity such as mouse clicks and keyword events. These events can be easily played back.

Package management (Composer)

Composer is a package manager for PHP. It allows keeping track of third party PHP libraries used in a project. To use Composer you have to create a file called composer.json in your project directory. Then you have to add the location of the PHP libraries in composer.json. After that enter the command: composer install. This will install the packages mentioned in the composer.json file.

The packages are by default installed to the vendor directory. If the package is updated by its maintainer you can update it in your project by running the command: composer update.

By default composer uses the PHP libraries on the packagist website, but it can be configured to use other sources as well such as phpclasses.org or Github. Composer has an autoload feature that allows all Php libraries to be included in your project using a single include command. e.g include(“autoload.php”) will include all the libraries managed by composer.

Version Control (GIT)

Git is a popular distributed version control system. I used it for keeping track of the different file versions. After completing a feature I saved the changes to the version control repository.

Issue tracking (Mantis BT)

Mantis is a popular free issue tracking system that allows keeping track of issues. I use it for working wit the client. In my bog post titled: Tracking Issues with Mantis Bug Tracker, I described my experience with configuring and using Mantis.

Conclusion

There are many other Software Development methodologies such an continuous integration, agile development, lean development etc. I think Its important to use the most suitable methodology for your project instead of using the methodology that is most popular. I learned a lot from the HQFollowBack project. It improved my knowledge of Software Development.

How to become an Itunes Affiliate

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

The Itunes Affiliate program allows website owners to earn commission by adding links to Itunes products to their websites. Website owners can link to millions of songs, movies, tv shows, books and apps on Itunes and earn commission from the sales

The Itunes Affiliate program is a useful monetization opportunity for websites that provide content such as books, movies and apps. The website has to link to content on Itunes. Apple gives a share of the sales to the website owner. Apple provides different ways of linking to its content, including widgets, banners and badges.

For example to add an Itunes badge to your website, you can use the Itunes Link Maker tool. It allows users to search the Itunes API from a browser. The API returns list of products that match the given criteria. Each product has an embed code that you can add to your website.

If you want to automate the process of adding Itune Badges to your website, then you can use the Advanced Search API. This is a RESTful API that returns results in JSON format. You can use the API from your favorite programming language such as PHP. The search results of the API does not include embed code for the banner but it does include a track id that we can add to the banner embed code. In this way we can fully automate the process of adding Itunes Badges to a website

After you have added the embed code for a badge to your website, you need to add an affiliate id to the code. You can get this id by applying for the Itunes Affiliate program.

Once you have your Itunes Affiliate id, you have to add some JavaScript code to your website that turns all the Itunes links on your website into affiliate links. You can get this JavaScript code from the Auto Link Maker tool. After you have added this JavaScript code to your website, you can start earning commission.

Conclusion

Apple provides useful information on its website on how to join the Itunes Affiliate program. Hopefully this article gave you some insight on how to make use of the Itunes Affiliate program.

Installing LXC containers on Debian Jessie

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Containers are light weight virtual machines that can be used to isolate processes. Containers implement operating system level virtualization. This means that all containers running on a host will share the kernel of the host operating system.

The main difference between Containers and Hyper-Visors such as XEN and KVM is that the later implements hardware level virtualization. This means a host system running XEN or KVM can run virtual machines with different kernels.

The main container technologies in use today are Docker and LXC/LXD. LXD is a refinement of LXC and is based on LXC. Until recently OpenVZ was a popular choice for containers. It has now been replaced by containers based on CGroups.

CGroups is a new feature of the Linux kernel that allows process isolation. Both Docker and LXC are based on CGroups. Since the Linux kernel now has built in support for containers, old container technologies such as OpenVZ have been largely replaced by Docker and LXC.

Installation

The installation of LXC on Debian Jessie is very simple and only requires the command: apt-get install lxc. The LXC home page has useful information on how to get started with LXC

Creating new containers

To create a new LXC container, we have to run the command: lxc-create -t download -n your-container-name. This will display a list of options for the operating system. Once you have selected your operating system, the template for your selected operating system will be downloaded and the new container will be created.

Using containers

Here are some commands for using containers:

To start your container, you have to run the command: lxc-start -n your-container-name -d.
The command lxc-info -n your-container-name displays information about the new LXC container.
To enter the container shell you have to use the command: lxc-attach -n your-container-name.
To stop your container enter: lxc-attach -n your-container-name.
To destroy your container enter: lxc-destroy -n your-container-name.

Setting up networking

The Debian Wiki LXC page covers the basics of LXC container management including setting up networking and mounting external folder shares. According to the Wiki, there are three main options for configuring networking on LXC. Using a simple Bridge, Vlan networking and setting up a network based on LibVirt.

Conclusion

LXC containers are very easy to use. They allow easily testing new software without interfering with the main host. They are also light weight and do not take lot of server resources. LXC containers can be used to logically sub divide your computing work into groups. I use LXC containers for my website development work and find them very useful.

Installing W3C Markup Validator and Validator Nu on Debian Jessie

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

W3C Markup Validator and Validator.nu are two useful tools that can be used to validate HTML code. W3C Markup validator allows validating xhtml, html, xml, mathml and more. Validator.nu allows validating HTML 5 code. The W3C Markup validator is written in Perl, while the Validator.nu is written in Java. Both tools are actively being developed.

Both tools have well documented APIs. The API for W3C Markup Validator is based on soap while the API for Validator.nu is RESTful and can be accessed using HTTP GET or POST methods. Both tools can be accessed online from a web browser or from their respective APIs.

If you need to validate several documents then it makes more sense to install the tools locally. By installing the tools locally we can validate documents more quickly and reliably. In this article, I will describe how to install both tools on Debian Jessie.

Installing W3C Markup Validator

The W3C Markup Validator can be installed by following the instructions given on the W3C Markup Validation website. The website provides instructions on installing the W3C Markup Validator from a pre built package or from source.

Installing the pre requisites

I tried to install the W3C Markup Validator on Debian Jessie from package but the package was not available in the default repository. I decided to follow these instructions for installing the W3C Markup Validator from source.

The installation steps are fairly easy to follow. First you need to install the prerequisites which is the SGML parser called opensp and some Perl modules.

The install guide mentions installing either several Perl modules one by one or installing Bundle-W3C-Validator perl module. I installed the Bundle-W3C-Validator bundle module but I still had to install the Perl module: SGML::Parser::OpenSP

Downloading the source code

Next you need to download the source code for the W3C Markup Validator from Github. The W3C Markup Validator requires a web server ideally Apache. The validator source code is written in Perl and requires mod_perl Apache module. Installing mod_perl from source was the most difficult step in the installation.

Installing mod_perl

The Installing mod_perl 2.0 guide describes how to install mod_perl from source. Mod_perl requires perl headers which can be obtained by installing the libperl-dev package using apt-get. After the mod_perl source code has been downloaded, we need to generate the make file using the command:

perl Makefile.PL MP_APXS=/usr/local/apache2/bin/apxs

I got following error when running the command:

Your Perl is configured to link against libgdbm,
but libgdbm.so was not found.
You could just symlink it to /usr/lib/x86_64-linux-gnu/libgdbm.so.3.0.0

I fixed the error by installing the libgdbm-dev package. I ran the command again but got this error:

Reading Makefile.PL args from @ARGV
   MP_APXS = /usr/local/apache2/bin/apxs
no conflicting prior mod_perl version found - good.
Using APXS => /usr/local/apache2/bin/apxs
[  error] Unable to determine server version, aborting.
[  error] Invalid MP_APXS specified?

I fixed the error by installing the apxs package using the command:

apt-get install apache2-dev

The apxs package is a tool for building extensions and modules for Apache 2.4.

The apxs binary was installed in /usr/bin so I had to update the command to:

perl Makefile.PL MP_APXS=/usr/bin/apxs

After that the package had to be built using make command. After that make test command had to be run for running the package tests. If the tests pass then the mod_perl can be installed using the command make install. The last command copies the mod_perl.so library file to the Apache module directory. The module can be enabled using the command a2enmod.

During installation of mod_perl the make test command failed with some errors. I decided to post the error to the mod_perl mailing list. After mod_perl was installed, I followed the rest of the installation instructions for the W3C Validator tool.

I had to first copy the validator.conf configuration file to /etc/w3c/ folder and then adjust the folder paths in the file. Next configure a virtual host for the W3C Markup Validator tool. After that I restarted Apache and was able to access the W3C Markup Validator from a browser.

Installing Validator.nu

Validator.nu allows validating Html 5 documents. Its installation is much simpler. The installation steps are given on the Nu Validator Checker website.

First we need to install the pre requisites which are Git, Python and JDK. The python version should be 2.7 or greater. The JDK version should be 8. JDK 7 did not work for me. Installing JDK 8 on Debian Jessie requires the backport repository for Debian Jessie. This can be installed by following the Debian Backport Installation instructions.

Building validator.nu from source

After the pre requisites have been installed you need to download the source code for the validator.nu from Git Hub. Once the source code has been downloaded, it needs to be built using the command:

python ./build/build.py all

This command starts a web server that listens on port 8888. The Validator.nu should now be accessible on http://localhost:8888. The validator.nu source code includes a vnu.war file that allows the tool to be deployed within a Servlet container such as Tomcat. There is also a vnu.jar file that allows HTML documents to be validated from the command line.

The W3C Markup Validator and Validator.nu are separate tools but can be used from the interface for the W3C Markup Validator. To use both tools from a single interface, we just need to uncomment two lines at the bottom of the file /etc/w3c/validator.conf.

Conclusion

The W3C Markup Validator and Validator.nu are useful tools for web developers that allow validating HTML markup. Both tools provide useful APIs that allow validation of HTML documents from any programming language. The APIs can be used in unit tests for web applications.

Installing and configuring Redmine

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Redmine is a web based project management and collaboration platform. It is based on the Ruby programming language and the Ruby on Rails framework. Its user friendly and easy to install and configure.

It has useful features such as project management, user and role management, issue tracking, wiki, forum, files and document management, time tracking and more. It is well suited for managing Software Development Project but it can easily be used to manage other projects. In this article I will describe my experience with installing and configuration Redmine.

Installation

Installation of Redmine is easy and is described in the Redmine Install Guide. The Redmine wiki has installation guides for different platforms. I followed the guide for installing Redmine from source on Linux. I used MySQL for the database.

The installation worked without problems. The only issue was installing the Redmine in a sub folder. The Url had to look like https://companyname.org/companyid/projects/. Sub folder installation of Redmine is described in the guide: HowTo Install Redmine in a sub-URI. It required adding following lines to the end of the config/environment.rb file:

RedmineApp::Application.routes.default_scope = "/companyid/projects"
Redmine::Utils::relative_url_root = "/companyid/projects"
ActionController::Base.relative_url_root = "/companyid/projects"

After that I had to create the folder /remine_install_path/companyid. Then I had to add a symbolic link to the public folder using the command:

ln -s /redmine_install_path/public /redmine_install_path/companyid/projects

Next I had to add the following line to the Apache virtual host configuration for the Redmine website: RailsBaseURI /companyid/projects. I had run the Redmine website behind Apache web server using the libapache2-mod-passenger module.

After the Redmine was installed, I had to login to the Redmine and give the full path of the Redmine url under Settings->General->Website Name. I set the path to: companyname.org/companyid/projects. The protocol field had to be set to https.

Configuration

Redmine provides a lot of features out of the box related to project management, issue tracking, document management etc. However sometimes we may required additional features or customization that are not provided by the default Redmine installation.

Redmine supports plugins and themes which can be used to enhance the functionality of Redmine. Redmine plugins can be downloaded from the Redmine Plugin Repository or from Github. Redmine plugins can add useful features such as allowing customization of the Redmine menu, Invoicing, Contact Management, adding custom JavaScript and CSS to the website, Issue Templates, Product management, Frequently asked questions, Agile related features and more.

Installation of Redmine plugins is a bit tricky. Each plugin has its own installation guide. Generally Redmine plugins are installed by decompressing the plugin file, then running the command:

bundle install

from the Redmine root directory. After that the command:

rake redmine:plugins:migrate RAILS_ENV=production

is run to import the plugin configuration information to Redmine. Next we may need to restart Redmine using the commands:

touch tmp/restart.txt
service apache2 restart

Installation of Redmine themes is very simple. It requires decompressing the theme file in the public/themes folder. Next Redmine needs to be restarted. The new theme should then show under Settings->Display->Themes.

Conclusion

Redmine is an excellent tool for managing projects. With the help of plugins and themes it can be transformed into systems that provide a wide range of features such as content management, account management, customer management, agile project management and more.

Integrating Rocket Chat and Trello

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Rocket Chat is an excellent open source collaboration tool. It is similar to the close source commercial application Slack. Trello is a commercial Kanban board that allows managing tasks and task lists. It has an attractive interface with many features and a well documented API.

The Trello API is easy to use and allows Trello to be integrated with other applications. In an earlier article I described how to install Rocket Chat on Ubuntu server. In this article I will describe how to integrate Trello with Rocket Chat.

Using Web Hooks for integration

We can integrate Rocket Chat and Trello using web hooks. Both tools provide web hooks as part of their APIs. The Trello API provides a sandbox from where we can test the web hooks

The Trello API considers each object as a model. For example a card, checklist, board etc are called models. Each model has a unique id. A web hook is attached to a model by giving it the id of the model. Once a web hook has been attached to a model, it will fire each time the model is updated. For example if we create a web hook on a Trello card and then delete the card, the web hook attached to the card will fire. Each web hook has a callback url that is called when the web hook fires.

Creating incoming and outgoing web hooks in Rocket Chat

The Rocket Chat administration panel has a section called Integrations, from where we can add incoming and outgoing web hooks. The Rocket Chat documentation describes how to create incoming and outgoing web hooks.

To create an incoming Web Hook in Rocket Chat we have to give the name of the channel that uses the web hook, the name of the user of the web hook and the script that should be run when the web hook is called. An outgoing web hook is created similarly.

The Rocket Chat documentation provides sample scripts for both incoming and outgoing web hooks. This script is run when the web hook is called. After an incoming web hook is created, the Rocket Chat generates a URL that can be used by other applications. For example when an application calls the incoming URL, Rocket Chat will run the script that we specified when creating the incoming web hook. The script can do some processing and return the result as a message. This message is displayed to the channel users.

After we have created an incoming web hook in Rocket Chat, we have to copy the generated URL and use that as a callback URL for the Trello web hook. So each time a Trello object is updated, the Rocket Chat URL is called automatically. When this URL is called, the script associated with the URL is run on the Rocket Chat server and the result is displayed to the end user.

The Trello web hook passes parameters to the callback URL so the script for the incoming web hook can use the parameters. e.g for displaying the name of the Trello card to the user. The incoming web hook basically displays information from an external application to the channel users.

When we create an outgoing web hook, we have to give one or more URLs that should be called when the web hook is fired. The outgoing web hook is fired by Rocket Chat when the user enters certain terms. For example a command for listing all Trello cards in a certain list. When the user enters the command, the script associated with the outgoing web hook is called.

This script can access the URLs associated with the outgoing web hook. Depending on the user choice, the outgoing web hook can be fired each time the user enters a message or when a certain text appears in the message.

The script given in the outgoing web hook is specified as a class with two functions. One is called prepare_outgoing_request and is called each time the user enters text. The second one is called process_outgoing_response and is called when the URL given in the outgoing web hook is called. The process_outgoing_response function gets the response from the URL. When the first function prepare_outgoing_request is called, it can either call the URL associated with the web hook or it can display help information to the user. For example on how to use the command.

Conclusion

Rocket Chat is well documented and actively supported. It is an interesting open source project with many possibilities for integration.

Installing Rocket Chat and Wekan on Ubuntu Server

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Rocket Chat and Wekan are useful online collaboration tools. Both tools are open source and can be downloaded and installed for free.

Rocket Chat is a chat application similar to Slack, while Wekan is a Kanban application similar to Trello. Rocket Chat and Wekan are easy to use and can be used for collaborative project management. In this article I will describe how to install and configure Rocket Chat and Wekan.

Both Rocket Chat and Wekan are based on the open source Meteor JavaScript framework. Both applications are well documented and are supported by an active community.

Rocket Chat Installation

Rocket Chat consists of a service and clients that connect to the service. The service has several installation options. It can be installed on the Rocket Chat Cloud or on a Platform as a Service (PAAS) such as AWS, Heroku, Sandstorm.io, Automation Tools etc. It can also be installed as a Docker Container or manually from source code. Rocket Chat clients are available for several platforms including iPhone and Desktops clients for Windows and Linux. The easiest way to access the Rocket Chat service is from a web browser, which requires no installation.

I had installed the Rocket Chat service from source. The Rocket Chat documentation has guides for installing from source on different operating systems. I followed the Ubuntu server installation guide. After installing the Rocket Chat service, I was able to access the service from a Linux desktop client and web browser. Both clients worked well without issues.

Rocket Chat has following dependencies: nodejs, mongodb, curl and graphicsmagic. It requires nodejs version 0.10.43 or higher. nodejs can be installed using Node Package Manager (npm). The main issue I had was installing Rocket Chat over https behind Apache web server. Furthermore the installation had to use a subfolder url. Fortunately the Rocket Chat documentation covers this installation case very well.

Rocket Chat Installation Issues

After installing Rocket Chat I had problems with getting the Jitsi.Meet video chat function to work. Jitsi Meet is an open source WebRTC application that allows video conferencing. Jitsi Meet video conferencing is integrated into Rocket Chat channels and direct chat. I had problems with getting the Jitsi Meet video function to work. I had to add following rules to Apache virtual host configuration:

ProxyPass /packages http://localhost:1000/chat/packages
ProxyPassReverse /packages http://localhost:1000/chat/packages

After that the Jitsi Meet feature started working but only on Firefox browser. Chrome browser gave a permissions error. I posted the error on the Rocket Chat GitHub issue tracker (Issue 3973, Issue 3952 ).

Customizing Rocket Chat

Rocket Chat can be easily customized by replacing the logo, favicon and login page banner. The favicon can be uploaded in different sizes and types including SVG. Rocket Chat also allows adding custom CSS code. I used it to style certain parts of the login screen.

Wekan Installation

Wekan like Rocket Chat is based on meteor. Installation of Wekan is well documented on their website. Wekan can be installed from Sandstorm, Docker, Binaries or Source Code. I chose the last option, i.e installing from source code.

Wekan Installation Issues

I had to install Wekan over https behind Apache web server. The URL had to be a sub folder. This type of installation is not documented in the Wekan Wiki. The installation was mostly successful, but had one bug. When I clicked on a card it showed a “Page not found” error. I decided to post the issue on the Wekan Github issue tracker (Issue 666). Another drawback of Wekan is that it does not have an admin section from where we can manage users. Its a useful tool but does not have as many features as Rocket Chat.

Startup scripts for Rocket Chat and Wekan

Both Rocket Chat and Wekan are started in the same manner. i.e using the command: node main.js. This command starts a server that listens on the given port and URL. The port, URL and database information is given to the application as environment variables. To manage the Wekan and Rocket Chat services more easily, I decided to use the forever service.

The forever service allows managing services based on NodeJs. On Ubuntu it can be installed using the command: apt-get install forever. Installation and configuration of the forever service is described in Step 4 of the guide: How To Install, Configure, and Deploy Rocket.Chat on Ubuntu 14.04.

Conclusion

Both Rocket Chat and Wekan are useful tools that can be used for collaboration and project management. Integration of these tools would be a great feature that has been much requested.

Load testing websites using Apache Bench, Apache JMeter and Loader.io

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Load testing a website refers to the process of testing the number of concurrent users that the website is able to support. Whether or not your website attracts lots of visitors, it is useful to know how many users the website can support.

Recently I had the opportunity of load testing a website. The tools I used for load testing the website were: Apache Bench, Apache JMeter and the online service loader.io.

Benefits of Load Testing

Website load testing is a very useful practice that gives you an idea of how many users your website can support without experiencing a decrease in performance. If your website is getting a lot of visitors and its performance is starting to degrade, then you may want to improve its performance by using HTTP caching or upgrade the web server.

If you do not have control over the hosting of your website, then you can upgrade your hosting plan or change your hosting provider. In most cases efficient use of HTTP caching can fix most website performance problems.

Apache Bench

Using Apache Bench

Apache Bench is a command line tool for measuring performance of the Apache webserver. It is bundled with the standard Apache distribution and is run using the command ab. On Ubuntu in order to use the ab command, you need to install apache-utils.

Apache Bench is single threaded which means it only uses one operating system thread so it can be a bit slow at times. The following command load tests the website http://dev.example.com using Apache Bench:

ab -n 500 -c 100 http://dev.example.com

The command sends 500 requests to the web server in total with a concurrency of 100.

The result showed that the average time taken for a request was just 25 seconds for a batch of 100 requests, which is just 0.25 sec per http request. The ab utility only fetches the main page and not the resources within the page such as images, CSS, JavaScript files etc.

Generating graphs from output of Apache Bench

We can export the results of ab command to a format suitable for GNUPlot using -g outputfile_name. This can give us a nice graph showing how the website response time changes with the number of requests. We can use the online version of GNUPlot for quickly creating a graph of website load times. This is described in the article titled “Apache Bench and Gnuplot: you are probably doing it wrong”.

Loader.io

loader.io is an excellent online load testing service that has a great free plan. Their free plan allows 10,000 clients per test but only 1 target host. The free plan also allows scheduling test result execution and notification of test results using web hooks. I ran a test on loader.io for a website and it gave the following results:

Apache JMeter

Apache Jmeter is a well known load testing tool. It is used by most online load testing service providers. The Apache Jmeter tool is written in Java and has lots of options. It can actually be used to load test any application over a TCP/UDP port such as DNS, LDAP, SMTP, Web servers etc.

It is usually run from a GUI but can also be run from the command line. The Apache Jmeter user manual is an excellent resource for getting started.

To start using Apache Jmeter, you have to first download it from the Apache JMeter site. Once you have downloaded the file, extract it to a folder and if you are on Windows run the file jmeter.bat or jmeter.sh if you are on Linux. This will start the Jmeter GUI.

To start testing a website, create a new test plan using the “Building a Web Test Plan” template. This creates a new test plan with some pre configured components. You have to edit some of the values such as the name of your web server and the number of users to use during testing.

JMeter allows you to easily change the graph component and the user parameters. The “Graph Results” listener that is part of the test plan does not give a very nice looking graph. To display a better looking graph you just have to add a new listener component such as “Aggregate Graph”.

Conclusion

Apache Bench, Loader.io and Apache JMeter are all excellent options for load testing your website. All three have support for integration with third party frameworks such as Jenkins Continuous Integration Server.

Load testing is a very useful method for testing your website. It gives us a very important performance metric which is the variation of web page load time with number of users.

Monitoring CPU and Memory usage on Linux

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

CPU and Memory monitoring is an important system administration task. It provides valuable information that can be used in resource planning and alert notifications.

For example if CPU and Memory usage is consistently high then the system administrator may want to consider moving his system to a server with more resources. He may also want to get a notification by Email or SMS so he can take action before his system is damaged.

Configuration

Linux based operating systems provide the virtual file system /proc. It contains information provided by the kernel, including information on CPU and RAM usage. The files /proc/cpuinfo and /proc/meminfo provide information on CPU and Memory respectively.

The contents of these files can be checked on a regular basis from a simple script. A notification email may be sent in case usage exceeds a given threshold.

Monitoring CPU

CPU usage can also be monitored with the top command which is part of most Linux distributions. The following bash command returns the CPU usage:

cpu_usage=$(top -b -d1 -n1|grep -i "Cpu(s)"|head -c21|cut -d ' ' -f3|cut -d '%' -f1);

Monitoring RAM

Memory usage can be extracted from the /proc/meminfo file using the following command:

# the total memory. the MemTotal row is extracted using awk. the second column of this row is then extracted and saved to total variable
total=$( awk '/MemTotal/ {print $2}' < /proc/meminfo )
# the available memory. the MemAvailable row is extracted using awk. the second column of this row is then extracted and saved to available variable
available=$( awk '/MemAvailable/ {print $2}' < /proc/meminfo )
# the formula for calculating percentage of available memory is piped to bc command. the output of bc command is saved to memory_available variable
memory_available=$(echo "($available*100/$total)" | bc)

CPU and Memory monitoring is very import on production servers. Some hosting providers like Google Cloud Platform (GCP) automatically shut down your server if the CPU and Memory usage appears suspicious.

Recently a server that I had been managing on GCP was suspended because of “suspicious activity”. The only thing that I could think of that could have caused the problem was ClamAV anti virus. It runs once a day and scans the server files for viruses. It caused a spike in CPU usage.

GCP showed a high CPU warning for the instance on the Google Console page. I did not take the warning seriously and as a result the instance was suspended. We had to pay 100 USD to restore our instance. I set up a bash script for monitoring the CPU and RAM usage. It runs every 15 minutes as a cron job. The script reported high memory usage but I did not take the error seriously.

That proved to be a big mistake because GCP again suspended our instance and we had to pay 100 USD to restore it. This time I checked the server again and found out that the server was running LXC containers and each container was running its own instance of the ClamAV anti virus.

This was not needed since the main server that ran the containers was already running the ClamAV anti virus and it checks the container files for viruses. After I removed the anti virus from the containers, the Memory and CPU usage patterns because normal and so far we have not received a warning from GCP.

Conclusion

CPU and Memory monitoring is an important system administration task that can prevent big problems for the system administrator. Although there are log monitoring tools and services such as GrayLog, Splunk and Google’s Log Stack, I prefer to use monitoring scripts that I find more dependable.

Responsive website design with Twitter Bootstrap

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Designing your website so it is accessible on all type of devices is a popular topic and is often referred to as Responsive Website Design. This blog post describes my experience of developing a website according to Responsive Website Design principles.

Optimizing your website for mobile devices

These days a lot of website visitors use mobile devices. Mobile devices are usually small with much less resources than a desktop computer. A website that loads normally on a desktop computer may load very slowly on a mobile device. For example some buttons may be too close to each other or may be to small to select.

There are different ways in which you can provide a good web browsing experience to mobile users of your website. You can construct a separate version of your website for mobile devices and redirect mobile users automatically to the mobile version.

Another option is to construct an application for mobile devices that can be downloaded from an App Store like Google Play. The drawback of these two approaches is that you need to maintain separate versions of your website for different users. My customer wanted a single website that works equally well on desktop computers and mobile devices. The solution was to construct the website based on Responsive Website Design principles.

Using the Responsive web framework, Twitter Bootstrap

My approach was to use a Responsive web framework for developing the website. I decided to use Twitter Bootstrap. Twitter Bootstrap is a very popular web framework based on HTML 5

It provides several components that work equally well on desktops and mobile devices. Downloading the Twitter Bootstrap library and using one of its examples as the layout for the website was simple

The Problem

The main problem was that the customer wanted the footer and left sidebar of his website to look like the ones on Digital Ocean.

The solution: Use browser detection code

The answer to our problems was to use code that detects the size of the web browser. Most modern libraries support browser detection. I used following browser detection options:

CSS Media Queries

CSS Media Queries are supported in CSS3. They allow us to use certain CSS statements depending on the dimensions of the device. They provide a very simple way of getting your website to look good on mobile devices. For example the following CSS Media Query code will cause the browser to use certain CSS classes if the width of the browser is 600px or less:

/** Condition */
@media screen and (max-device-width: 600px), screen and (max-width: 600px) {
/** CSS classes used if condition is true */
*[class=mobilewrapper]{width:100%!important; height:auto!important;}
*[class=w320]{width:320px!important; height:auto!important;}
*[class=split]{width:320px!important; float:left!important; display:block !important;}
*[class=left-sidebar-container]{display:none;}
*[class=center-sidebar-container]{display:block;}
*[class=toc_container_mobile]{display:block;}
*[id=comments-box]{background: #DCE2ED;padding: 7px;border: #ACB1BA solid 1px;margin-right: 0%;width:100%;}
}

In this way you can use different CSS classes depending on the width of the device. One of the main changes that HTML 5 introduced was to force the use of CSS classes instead of the style attribute. This makes it much easier to change the presentation of a website using CSS Media Queries.

Browser detection using JQuery

You can also detect the device width using JQuery. If you are using JQuery to create certain user interface elements, then you may want to use different JQuery code depending on the device.

The article titled Detecting Browser Window Size/Resize with jQuery has a good description of how to detect browser width using JQuery.

Browser detection using PHP

If you are using PHP and you want to use different server side code depending on the type of mobile device, then you can use the mobiledetect library. This is an excellent library written in PHP that detects the user agent that is trying to access your web page. For example you can use the following code to detect if the website visitor is on a mobile device:

// Include and instantiate the class.
require_once 'Mobile_Detect.php';
// Include and instantiate the class.
$detect = new Mobile_Detect;
// Any mobile device (phones or tablets).
if ( !$detect->isMobile() )

Other options for Responsive Website Design

Another useful library that can be used in Responsive Website Design is Modernizr. This is a JavaScript library that can be used to detect the HTML 5 and CSS3 features available to a visitor of your website. You can then change your JavaScript or CSS code depending on the features available to the user. For e.g if HTML 5 canvas features is available to a user then you can use canvas related code.

It is also a good idea to make your website accessible to visually impaired users. A large number of internet users are visually impaired and use helper devices such as Brail to access the internet. Design Websites For Blind/Visually Impaired, Deaf & Disabled Visitors and Tips and Tricks to Improve Web Accessibility are useful articles that describe how to make your website accessible for visually impaired users.

You can check your website on your mobile phone or test it using online tools. Tools such as Google’s Mobile Friendly Test and the Responsive View in Firefox browser can be used for checking the accessibility of your website on mobile devices.

Conclusion

Responsive website design is a useful skill that can greatly improve the accessibility of your website. Websites are becoming increasingly popular and it is important for web masters to design websites that are accessible to all users.

Running LXC containers on a Google Compute Instance

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Google Cloud Platform is a Google product that provides services related to Cloud Computing. Services provided by Google Cloud include Compute, Storage and Databases, Networking, Big Data, Machine Learning and more.

The Compute service includes Compute Engine, App Engine and Container Engine. Compute Engine allows management of virtual machines. App Engine is a powerful platform for building scalable web and mobile applications. Container Engine allows management of Docker Containers.

This article describes how to run multiple websites on Google Cloud using an approach that is based on LXC containers.

Running multiple applications in the cloud

If you want to run several applications in the cloud, one option is to run the applications on a single Docker Container or a single Compute instance. But what if the applications are based on different platforms such as (Linux, Apache, MySQL, PHP) LAMP stack and (MongoDb, Express, AngularJs, NodeJs) MEAN stack. You may want to keep your LAMP and MEAN applications separate. One option is to run the applications on multiple compute instances.

If you have a small budget then this may not be possible. Another option is to run the applications within separate containers running on a single compute instance. LXC is a good choice for containers because it is present in the default package repository of most Linux distributions. It is also lightweight and consumes very little system resources.

Configuring networking for LXC containers

The installation of Linux Containers (LXC) is straight forward and is described in Installing LXC containers on Debian Jessie. There are different options for configuring networking for LXC containers.

The approach I used is based on libvirt and is described in the Debian Wiki article on LXC networking. The libvirt network creates a bridge networking device on the main host. It also runs a DHCP server and a service called DNSMasq. The DNSMasq service provides DNS services to the containers. The containers run DHCP client services which are used to fetch the ip address from the DHCP server. The ebtables bridge firewall is also used. The main steps involved are as follows:

Installing Libvirt. This can be done with the command apt-get install libvirt-bin
Configuring the default network. This involves editing the default.xml configuration file for the default libvirt network. The file requires the start and end DHCP ip addresses of the LXC containers and the ip address of the bridge interface
**Start the libvirt default network.**The libvirt default network is started using the command virsh net-start default. Use the ifconfig -a command to verify the existence of the bridge interface
Edit the container network configuration. Edit the file: /var/lib/lxc/a_container/config, where a_container is the name of your container. It should have a networking section similar to the following:

# Network configuration
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = virbr0
lxc.network.hwaddr = 00:FF:AA:00:00:03
lxc.network.ipv4 = 0.0.0.0/24

Set the containers to auto start. To auto start the containers each time the host server reboots, each container’s configuration file should have following lines:

# Autostart
lxc.start.auto = 1
lxc.start.delay = 5

The line lxc.start.delay is used to start the container after a delay of 5 seconds. This is done so that there is an interval between the starting of containers. For some reason this did not work for me and all the containers did not start on boot, so I had to add following to /etc/rc.local:

lxc-start -n a_container -d

Running multiple websites behind Nginx proxy server

It is possible to run multiple websites on different containers. These containers can run on a single host. The websites can be served by a single instance of Nginx running on the main host.

The Nginx web server acts as a reverse proxy server and serves content from the different websites running on the containers. To configure Nginx as a reverse proxy server for a website the following configuration file can be used:

server {
  listen 80;
  listen [::]:80;

  root /var/www/html;

  # Add index.php to the list if you are using PHP
  index index.html index.htm index.nginx-debian.html;
  server_name hostname-of-your-website;

  location / {
    proxy_set_header Host $host;
    proxy_pass http://private-ip-address-of-your-container$uri$is_args$args;
  }
}

The above configuration file should be created inside the /etc/nginx/sites-available folder. A symbolic link to the file should be added to /etc/nginx/sites-enabled.

Conclusion

This article described an approach for running multiple websites on a single Google Compute Instance. It suggests running the websites on different containers and accessing the websites through a single instance of Nginx web server. This approach can easily be used on other Cloud Computing platforms such as Amazon Web Services (AWS) and Microsoft Azure.

Some excellant Wordpress Plugins

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

This blog post is about my experience with using WordPress plugins. WordPress is an excellent blogging platform that is used by millions. It has an enormous number of plugins that are maintained by people all over the world. You can find a WordPress plugin for just about any function you can imagine.

WordPress has evolved from a blogging platform to an application platform, content management system and more. Plugins such as Buddypress can transform your WordPress into a full featured online social community or even company intranet.

WordPress Plugins

I found the following WordPress plugins useful:

Wordfence Security This is an excellent plugin that keeps your WordPress secure. With increase in popularity, WordPress has become the target of hackers and bots. There are several viruses that target WordPress core and WordPress plugins.

Wordfence scans your WordPress files and emails you if it finds a suspicious file. WordFence compares the WordPress files with the ones from wordpress.org. If it sees a changed file, it sends a notification email. It also emails you when someone logs in to your WordPress site.

Wordfence also sends newsletters to its users by email. These contain important security related information for WordPress such as new WordPress viruses. One of their newsletters that I found interesting is titled “WordPress Security: Nulled Scripts and the CryptoPHP Infection”.

I recently had to recover a hacked WordPress website and found “How to Clean a Hacked WordPress Site using Wordfence”, from Wordfence very useful.
Wordpress SEO

This is an excellent plugin that can help you boost your rankings on search engines. It lets you specify the meta keywords and description for each blog post. This allows you to set the blog information that would appear on search engines like Google.
WP Slimstats

This plugin collects and displays statistics about your blog posts, such as the number of visitors, duration of visits, number of page views etc. There are other analytic plugins for WordPress but I found this one quite useful since it integrates well with WordPress. It provides a nice dashboard from where you can view detailed statistics about your website visitors. One drawback of this plugin is that it can slow down your website.
Enlighter - Javascript based syntax highlighting

This is a useful plugin for high lighting source code. If you are a developer like me and want to share code snippets then you will find this plugin useful. It displays your code in a nice highlighted box. You can change the style of the box. The plugin supports around 15 programming languages.
Shareaholic | share buttons, analytics, related content

This plugin allows you to add share buttons on your blog articles. It allows your site visitors to share your content over different social networks. It currently supports several social networks including Twitter, Facebook, Linkedin and Pinterest. It even shows you how many users shared a particular blog.
Google XML Sitemaps

This plugin allows you to submit your blog posts to Google, Bing, Yahoo and other search engines. It does this by automatically generating sitemap files. It also automatically informs search engines in case blog posts change.
Infinite Scroll

This plugin replaces the next/previous links on your blog with automatic scrolling. When the user reaches the end of the page, the next set of blog posts are automatically displayed. The user can view all your blog posts on one page.
NextGEN Gallery by Photocrati

This plugin allows you to embed photo galleries inside your blog posts. It allows fine grained control over gallery style. It also allows setting meta information for gallery images.
NextScripts: Social Networks Auto-Poster

This plugin allows you to submit your WordPress blog post to a large number of social networks. e.g Twitter, Linkedin and Facebook. The plugin has excellent documentation on how to connect social network accounts to your WordPress blog. It also gives a lot of options for posting to social networks. e.g you can choose to include images in your twitter posts.

Conclusion

WordPress has thousands of plugins. You can find several plugins that perform a required functions. e.g there are dozens of plugins for sharing on socials networks. Finding the right plugin for the job usually involves trial and error. You may have to try out several plugins before finding the one that best matches your requirements.

Tuning WordPress performance using Varnish

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Recently I had the opportunity of on tuning website performance using Varnish. Varnish is called a Caching HTTP Reverse Proxy. This means it sits between your end user and one or more back-end servers such as Apache or Nginx and functions as a cache for HTTP traffic. Varnish can also function as a back-end server for a front-end such as Haproxy. This is the configuration in which we used Varnish.

Varnish is an excellent option for busy websites. It is a very fast and efficient caching server that can greatly reduce the time taken for serving requests. Of course to get maximum benefit from Varnish we need to configure and tune it according to our needs

Installing Varnish

Varnish online documentation is complete and covers all aspects of using Varnish from installation to configuration and troubleshooting. Installation for Redhat based systems is simple and only requires adding the varnish repository to yum and running yum install varnish. This will install the latest version of varnish on your server.

The latest version of varnish is 4.0.x. The 4.x branch of Varnish is very different from the 3.x branch so Varnish configuration for 3.x will probably not work on 4.x. Its also difficult to find configuration samples for 4.x branch. If you need help with tuning Varnish you should read their documentation and their online support forum. Changes from the 3.x branch are documented on the Varnish website.

Basic Configuration

Once we installed Varnish on our Centos 6 server we configured it to listen on port 80 and configured Apache to listen on port 8008. To change listen port of Varnish we set the VARNISH_LISTEN_PORT=80 in the main Varnish configuration file /etc/sysconfig/varnish.

To change Apache port to 8008 we had to edit /etc/httpd/conf/httpd.conf and add port 8008 to the Listen directive.

After that we changed the Varnish backend port to 8008. This tells Varnish to communicate with Apache over port 8008. To do this we set **.port = “8008” **in /etc/varnish/default.vcl and restarted Varnish using service varnish restart..

Varnish can store its content in memory or in files. To configure Varnish so it saves content in memory we had to set

VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"

in /etc/sysconfig/varnish. Thats all that is needed for getting up and running with Varnish.

Serving content over Varnish

We had to tune and configure Varnish so it allows delivering content from our custom Content Delivery Network. We also had to configure Varnish so it supports WordPress. To do this we had to edit the main Varnish configuration file: /etc/sysconfig/varnish and the default varnish configuration file /etc/varnish/default.vcl..

Adding Content Delivery Network (CDN) server to Varnish

To add our Content Delivery Network (CDN) server to Varnish we did following:

Edited /etc/varnish/default.vcl and added following lines below the backend statement:

backend cdn {
  .host = "192.168.1.10";
  .port = "8000";
}

192.168.1.10 is the IP address of the CDN server. 8000 is HTTP listen port of the CDN server.

Restarted Varnish
Updated the DNS record for the CDN host name so it points to our Varnish server. Once we did that our CDN content was being served over Varnish.

Serving WordPress over Varnish

In order to serve our WordPress website over Varnish we had to add specific configuration information for WordPress to /etc/varnish/default.vcl. This file contains configuration information in a language called “Varnish Control Language” or VCL.

Using this language we can configure Varnish for various tasks. The VCL configuration file contains sub routines that correspond to different stages of request execution in Varnish. These sub routines are documented in the default.vcl files. They are:

vcl_recv. This is called by varnish just before it checks if the request is in cache or not. In this sub routine we can clear cookies from the request or rewrite the request.
vcl_backend_response. This is called just before Varnish reads the HTTP response headers from the back-end. In this sub routine we can clear cookies that are set by the back-end.
vcl_deliver. This is called just before the response is sent to the client. In this sub routine we can do basic accounting tasks or modify the final object before it is delivered to the client. For example we can set a HTTP response header indicating if the request was a cache hit or a cache miss.

Adding HTTP headers to Varnish output

If you have multiple Varnish back-ends. For example you have one HAProxy front-end that load balances between multiple Varnish back-ends, then it would be useful to know which Varnish back-end served a request. You can do this be using the following vcl code in vcl_deliver sub routine.

sub vcl_deliver {
resp.http.X-Server= "dev.example.com";
}

It sets the server name to dev.example.com. You can view the HTTP headers added by Varnish such as cache hit/miss, cache duration etc from Google Chrome’s Developer Toolbox.

Download the VCL

Varnish has some useful VCL code for WordPress but it only works with Varnish 3.x. I had to modify the code so it works with Varnish 4.x.

Varnish plugin for WordPress

WordPress has a useful plugin for Varnish called Varnish HTTP Purge. It allows WordPress admin users to remove or purge pages from the Varnish cache.

Securing Varnish

One of the tasks that are commonly carried out in Varnish is purging content from the Varnish cache. By default Varnish allows purging content from any host. This can create a security issue, so its best to configure Varnish so it allows purging content from local host or specific IPs.

To do this we need to create an Access Control List in Varnish that defines which hosts are allowed to purge Varnish cache. The following VCL statements achieved this:

Place following after the back-end statements:

acl purge {
  "localhost";
  "127.0.0.1";
}

Place following inside vcl_recv sub routine. It checks the HTTP request method. If the method is PURGE and the IP from where the purge request is made is supported by the above ACL then the object is purged from the Varnish cache and the next VCL statement is executed. If the IP is not supported by the ACL then an error is shown to the user.

if (req.method == "PURGE") {
  if (!client.ip } purge) {
    return(synth(405,"Not allowed."));
  }
  return (purge);
}

Place following inside vcl_hit sub routine. vcl_hit is the function that is called when the object is found in the Varnish cache. If the request method is PURGE then Varnish returns success.

if (req.method == "PURGE") {    
  return(synth(200,"OK"));
}

Place following inside vcl_miss sub routine. vcl_miss is the function that is called when the object is not found in the Varnish cache. If the request method is PURGE and the object was not found in the cache, then the sub routine returns success since the object was not in cache anyway.

if (req.method == "PURGE") {    
  return(synth(200,"OK"));
}

You can also purge content from the Varnish cache using following command line: curl -X PURGE content_url. Make sure you issue this command from a host that is allowed by the ACL that you defined in your varnish VCL file.

The VCL configuration file of WordPress performs 2 main tasks. One is to allow purging of Varnish content from certain IPs. The other is to remove cookies from client HTTP requests and back-end HTTP response.

Varnish never caches content that has cookies. So any content that needs to be cached must be stripped of all cookies.

Of course you don’t want to cache content from WordPress admin pages so you need to add suitable VCL statements that exclude the admin pages. Another thing to note about Varnish is that it considers objects with different URL query strings as different.

So if you want to prevent a JavaScript file from being cached one option is to append a query string to the URL and change the query string each time the JavaScript file is updated and needs to be removed from the Varnish cache.

Checking errors

If you have an error in your VCL configuration file and you restart Varnish, then at least on Centos servers you wont see any descriptive error message. To see the error message you can start varnish manually from the command line using following: vanrishd -f /etc/varnish/default.vcl. This will show the errors in the VCL configuration.

Viewing statistics

Varnish also has built in tools for viewing statistics. These tools are:

varnishtop. It reads the varnish shared memory logs and presents a list of the most commonly updated log entries. e.g it can list ranking of requested documents, clients, user agents etc.
varnishhist. It reads shared memory logs and displays continuously updated histogram showing distribution of last N requests.
varnishtop. it displays different varnish counters such as hit rate, miss rate, information about storage, threads created, deleted objects etc.

Edge side includes

Varnish also has a feature called edge side includes or ESI. This allows different caching policies to be applied to different parts of a web page. For example if a web page includes a statement such as:

<esi:include src="/includes/some_content.html"/>

then that will cause Varnish to replace the include statement with the content. This can be used for example by a news web site for caching the list of top 10 news items.

Varnish is a very popular choice for use as HTTP cache. Many I.T tools provide support for Varnish. For example the monitoring tool, Munin has a plugin for Varnish that allows users to graph Varnish statistics.

Conclusion

While there are other HTTP caches such as Squid, I found Varnish to be an excellent option. It has many configuration options, its easy to configure, it has good statistics and log monitoring tools and is being actively developed.

Upgrading JWPlayer to version 7

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

JWPlayer is one of the most widely used video players. It has evolved from a simple video player into a video publishing platform. The latest version 7 of the player is a major upgrade from the last version. Websites that use the old JWPlayer 6 should upgrade to the latest version.

Changes in the new JWPlayer

The new version of JWPlayer contains many new features and has deprecated or removed many features in the old version.

The JWPlayer upgrade guide has useful documentation of the changes introduced in version 7. Briefly here are the changes in the new version:

The JW7 does not support the old XML based skin format of JW6. It uses a CSS based format for configuring skins
The JW7 can only be embedded on web pages with doc-type of HTML 5
The new JW7 uses RSS feed format for its playlists. It does not support the old XML format of JW6
The layout of playlists in JW7 is very different from JW6. The JW7 has built in support for playlists and does not support the horizontal playlist layout of JW6. The JW7 does support changing the playlist layout. By using CSS and JavaScript it is possible to design horizontal playlist layout for the JW7. I was able to design a horizontal layout for the JW7
The JW7 interface does not render with flash. It only renders with HTML 5. It does support flash media but renders with html5
The JW7 supports the new Analytics.js library for Google Analytics support. It no longer supports the old ga.js library
The JW7 no longer supports Internet Explorer 8
The JW7 requires a license for all versions of the player. A free license is available

Conclusion

The new JW7 is a major upgrade from the old JW6. Be sure to carefully read the documentation when upgrading to the new JW7

Using Youtube API for validating URLs

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

Google recently updated its Developer Console into a unified platform for accessing all of its APIs. The Google Developer Console not only provides a single location for accessing information about all Google APIs but also provides links to code samples and API SDKs. It also provides online tools for testing the APIs.

The YouTube API

The Google YouTube API is a comprehensive API for accessing YouTube features. The API has several methods grouped into resources. For example activity, thumbnail, video, watermark etc. Some of the methods require authentication using OAuth while others can be accessed without Authentication. However all API methods require an API Key. The API Key can be downloaded from the Development Console.

The YouTube API supports 4 operations. These are list, update, delete and insert. These operations are carried out on resources. My task was to check if a movie clip is available on YouTube. The most suitable way to do this was to use the list operation on the video resource.

The YouTube API is easy to download and use. Google provides excellent documentation on using the YouTube API. It also provides Client API Libraries and code samples for several languages.

For example to access YouTube API using PHP you have to first download the Google API PHP Client from Git Hub. After that you have include the autoload.php file in your code. Next you need to instantiate an object of type Google_Service_YouTube and call the relevant function.

To check for availability of a movie clip, you can use the function listVideos of the videos property. I used the following code for checking for availability of movie clips on YouTube:

/**
* Used to check if the given movie is available on youtube
*
* It uses youtube api and checks if given movie is available on youtube
* If a movie is not available then it returns false
*
* @param string $youtube_video_url the youtube movie url
*
* @return boolean $is_available indicates if the given video is available on youtube
*/
private function IsMovieAvailable($youtube_video_url) {
    /** The autoload.php file is included */
    include_once("autoload.php");
    /** Is available is set to false */
    $is_available = false;
    /** The youtube video id is extracted */
    $video_id = str_replace("https://www.youtube.com/watch?v=", "", $youtube_video_url);
    $DEVELOPER_KEY = $google_api_key;
    $client = new \Google_Client();
    $client->setDeveloperKey($DEVELOPER_KEY);
    // Define an object that will be used to make all API requests.
    $youtube = new \Google_Service_YouTube($client);
    // Call the search.list method to retrieve results matching the specified
    // query term.
    $searchResponse = $youtube->videos->listVideos('status', array('id' => $video_id));
    /** Each item in the search results is checked */
    foreach ($searchResponse['items'] as $video) {
      /** If the video id matches the given id then function returns true */
      if ($video['id'] == $video_id) {
        $is_available = true;
        break;
      }
    }
    return $is_available;
}

Conclusion

The YouTube API is very detailed and supports all functions that are carried out on the YouTube website. The API is easy to use and is well documented.

Validate your WordPress blog using WordPress XML-RPC API, Php Tidy and Nu HTML Checker

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

WordPress is a very popular platform for hosting websites. Over 74 million websites are based on WordPress. WordPress is also increasingly being used to develop complex web based applications.

There is a need for a testing system that allows testing applications and websites based on WordPress. Recently I had the opportunity of setting up a simple testing system for WordPress that validates WordPress blogs using the WordPress XML-RPC API, PHP Tidy and the online Nu HTML Checker. I will describe my experience in this blog post. I will also briefly describe a couple of ways in which you can test server side code of your WordPress plugins.

What is the importance of Validating your website

Its a good idea to ensure that your WordPress blog conforms to W3C standards. The World Wide Web Consortium (W3C) is the main international standards organization for the World Wide Web. The W3C standard is recognized by all major search engines. Websites that conform to its standards can be understood by search engines and web browses more easily.

Such websites have fewer performance problems and function more reliably across multiple devices. They are also sure to receive a higher ranking on the major search engines.

Validating your blog using Nu HTML Checker

The W3C validation website contains several tools that you can use to validate your website pages. These tools include:

Nu HTML Checker. It can checks HTML of any document. It supports html5
Link Checker. It checks for broken links on your page
RDF Validator. It checks and visualizes RDF documents
Mobile Checker. It checks the mobile friendliness of your website
Internationalization checker. It checks level of internationalization-friendliness of your website
RSS feed validator. Validator for RSS and Atom feeds
CSS Validator. Checks the CSS of your website
Markup Validator. Checks the HTML or XHTML of your website
Unicorn. Unified validator. HTML, CSS, Links and Mobile validator

The Nu HTML checker can be used to check any website for compliance with the new HTML 5 standard. I created a PHP script that uses the WordPress XML-RPC API for getting list of all pages, posts, custom posts on my blog. The XML-RPC API is very useful and you can use it to easily get list of all pages on your blog. Once you have the list of pages, you can test each page against different W3C standards.

For each post the script validates the post HTML using the PHP Tidy extension and the Nu HTML Checker. The PHP Tidy extension is a popular extension for PHP that allows repairing and formatting your HTML code. It supports HTML 4 validation but does not support HTML 5.

If a post cannot be validated the test ends with an error. I ran this script with PhpUnit and added it to Jenkins Continuous Integration Server. I scheduled the script to run every day, so the script tests all pages automatically and sends me an email if any of the tests fail.

Setting up projects in Jenkins is quite simple. You can refer to my earlier blog post titled Continuous integration and Project Management with Jenkins and Redmine for more information on how to add scripts to Jenkins.

Testing Server Side code of your plugins

Testing a WordPress website using PhpUnit is different from testing websites based on other platforms. WordPress has its own authentication system which prevents testing WordPress plugins and themes like normal PHP scripts. There are 2 main options for testing the server side code of your plugins:

WP-CLI. WP-CLI is a command line interface to WordPress. It can be used along with PhpUnit to unit test your WordPress plugins. The WP-CLI website has some useful documentation on how to test WordPress plugins.
Extend the WordPress XML-RPC interface inside your plugins. This method allows you to test your plugins using WordPress XML-RPC interface. Basically it allows you to call your test functions using WordPress XML-RPC interface.

Conclusion

WordPress is a very popular platform for websites. It is important to have a testing system in place that makes it easy to test WordPress based websites.

Ensuring that your website conforms to W3C standards is very useful. It not only improves your SEO score but also ensures compatibility of your website across different devices and browsers.

WordPress for small and medium businesses

Tue, 11 Apr 2017 00:00:00 GMT

Introduction

WordPress is an excellent platform for small and medium businesses (SMBs). Recently I had the opportunity of constructing the home page for a small business. I will share my experience in this blog post.

After working as I.T manager for a small business for a few years, I realized how important requirements are to the company. Requirements that seem insignificant to a technical person can be a matter of life and death for the company.

Its very important for the I.T support person to ensure company requirements are correctly and completely fulfilled. SMBs tend live on a day to day bases. Unlike large companies, SMBs have small capital and have to fight for survival. I.T support for SMBs needs to be alert and focused on the company requirements. Also the management of SMBs usually are not interested in technical concerns of I.T people. They just want the job done.

I.T Requirements of SMB’s

As web master for a SMB, I have learned that the company requirements tend to be very focused. At first the requirements seem very simple and can easily put off an “educated” I.T worker. However these requirements are very specific and meeting them correctly is no small task.

The home page for a SMB represents the SMB on the internet. It needs to have following features:

It should present the company’s work clearly
It should promote the company’s services and products
It needs to take advantage of business promotion features available on the internet, such as those provided by Twitter, Facebook and Linkedin
It needs to be easy to manage. For example adding new features to the Home Page should be easy

In my opinion, the most important feature of the home page is that it needs to be easy to manage. For this reason WordPress is my platform of choice for home page for a SMB.

Why use WordPress

WordPress is not only easy to administer, but it can also be extended easily. It has a large number of website themes to choose from. New features can be easily added as plugins. WordPress supports thousands of plugins. Some of the WordPress plugins I found useful are:

Slideshow Gallery plugin, for presenting product and service images attractively.
Making contact forms with WordPress is easy with the help of plugins such as Contact Form 7.
If you need to post your WordPress articles on social networks easily, you can use NextScripts: Social Networks Auto-Poster plugin.
If you want to make it easy for your website visitors to share information about your website, you can use the Shareaholic | share buttons, analytics, related content plugin.
WordPress also has some nice plugins for web masters such as WP Slimstat, WordPress SEO and Wordfence Security.

It can be easy to get carried away with the large number of WordPress features. WordPress has plugins for just about any purpose. You can get lost trying to find the plugin that meets the actual requirement. Its important to stay focused on the task at hand and not experiment with different plugins.

For a SMB website, its more important to just get the job done correctly. One may question the benefit of having thousands of plugins to choose from. I think WordPress has many great features but not all of them are suitable for a SMB website.

Conclusion

I think being easy to manage and update is the best feature of WordPress for SMBs. Having thousands of plugins and themes and being easy to customize is great but more suitable for custom applications instead of simple web sites for SMBs.

Customizing Redmine layout

Mon, 13 Mar 2017 00:00:00 GMT

The user interface of Redmine can be easily customizing using the redmine_custom_css and redmine_custom_js plugins.

To add a background image to the Redmine login page the following JavaScript code can be used:

$( document ).ready(function() {
  if ($("#top-menu").html().indexOf("Sign out") &lt; 0) {
    document.getElementById('main').classList.add("custom-background");
    document.querySelector("#content > h2").classList.add("hide");
    document.querySelector(".splitcontentleft").classList.add("float-right");    
  }
});

The above code checks if the “Sign out” link is present in the header. If it is not present, then it adds a CSS class called “custom-background” to the div with id “main”. It also does some minor formatting of the layout. The “custom-background” CSS class may be defined with the help of the redmine_custom_CSS plugin. It can be defined as follows:

.custom-background {
  background: #ffffff url(https://example.com/background.jpg) no-repeat scroll 50px 0 !important;
}

We can edit the Redmine top menu bar with the help of the redmine_custom_js plugin. For example the following code adds a new menu item to the top menu bar when the user is logged in:

if ($("#top-menu").html().indexOf("<ul>")>0) {
  var updated_list_items = "<li class='item-0'><a href='https:/example.com/'>Company Name</a></p></li>";
  updated_list_items     = updated_list_items + ($("#top-menu > ul").html());
  $("#top-menu > ul").html(updated_list_items);
}

The above code first checks if the list of menu items is present. If it is present, then it creates a new menu item and adds it to the front of the list of menu items.

Installing W3C Markup Validator on Debian Jessie

Mon, 13 Mar 2017 00:00:00 GMT

The W3C Markup Validator is a useful tool that allows validating html text. It can be used to check if a given web page or text conforms to the html 5 standard. The following points may be considered during installation of W3C markup validator:

local copy of validator is more reliable and easy to use for bulk use
W3C Markup Validator supports validation of html,xml, xhtml, html5, mathml and more
html5 validation requires validator.nu installation
deb file not available in default repository for Debian Jessie
W3C Markup Validator has to be installed from source. install guide given on: https://validator.w3.org/docs/install.html
install guide for mod_perl from source (https://perl.apache.org/docs/2.0/user/install/install.html
following perl module had to be installed: SGML-Parser-OpenSP
libperl-dev package required for mod_perl compilation (http://tara24.de/fix-configure-errorperl-development-headers-found/)
java 8 required for validator.nu
java 8 for debian jessie requires backport repository (http://backports.debian.org/Instructions/)

Responsive web design with HTML 5 and CSS 3

Mon, 13 Mar 2017 00:00:00 GMT

Responsive website design is a design practice which aims at designing web pages with a layout that works on all devices. It involves several techniques such as responsive images, grid layouts, fluid layouts, media queries and more. The Responsive Web Design - Introduction is a useful resource for learning about Responsive Web Design. Following are some frequently asked questions about responsive website design:

What is the difference between mobile website design and responsive website design

Mobile website design means design of websites for mobile devices only. Responsive website design means designing websites for all devices including tablet, mobile, desktop etc. Mobile website design usually means using special frameworks only for mobile devices. Mobile websites do not always look good on desktops but Responsive websites looks good on all devices.

What is the difference between view-port and screen size

view-port is viewing area of website. If you resize browser window then the view-port changes screen size is size of the device screen

So how do we apply view-port to resize

view-port is same as browser window, but without the toolbars and buttons. It is just the area where the website shows. Screen size is physical size of device window. When we say IPhone has 5 inch screen, it means screen size of IPhone is 5 inch. view-port is actual area where the website shows in the browser. When we resize browser window, the area where the website shows also resizes.

What is the key role of Graphics in Website design

In Website design, before HTML 5 and CSS 3 we had to use tools like Flash and Photoshop, but now we can use HTML 5, JavaScript and CCS 3 for creating good looking affects for websites. For example we can create graphs, charts, animation effects, games and more using just HTML 5, JavaScript and CSS 3

So HTML 5 and CCS 3 can create images

It depends on the type of images you need. But for most images you need a graphics tool like Photoshop or Gimp. Gimp is free and open source. You can also create graphs and charts with HTML 5 and CCS 3 and also animations and games.

Should I use Dreamweaver or Notepad++ for HTML 5 and CSS 3 ?

You may start with simple tools like Notepad++, then use more advanced tools like Dreamweaver or Gimp. It is usually better to first understand the concept using a simple tool.

What is responsive design

Mon, 13 Mar 2017 00:00:00 GMT

Responsive Design is a new trend in Web Design. It was started by Ethan Marcotte. Responsive Design is the design of websites that work well when viewed on a wide range of devices. For example a responsive website will look good on Iphone, Ipad, Tablets, Laptops and Desktops.

Responsive Design is not a single technique but a set of several CSS practices, such as proportional layouts, flexible design, flexible images, browser compatibility, pollyfilling, CSS media queries and more.

A responsive design should be used in most cases. However if the website in question needs to have special requirements specific to a device such as mobile phone, then responsive design is not a good option

A responsive website should look good across old browsers as well. However it is upto the web designer on how well old browsers are supported. Pollyfilling is a technique used by web designers for supporting old browsers. It requires using different HTML code for browsers that do not support certain HTML standards. There are several pollyfill libraries that provide fallback support to old browsers that do not support certain HTML 5 standards. e.g a video element may be replaced by an image or a paragraph saying the browser does not support the HTML 5 video element

The book Responsive Web Design with HTML5 and CSS3 is a useful resource for learning about Responsive Web Design.

What is the difference between viewports and device width

Mon, 13 Mar 2017 00:00:00 GMT

View port is the viewable area of a browser where the website is shown.

The device width is physical width of the device where the website is viewed. The view-port can be scaled using the HTML meta tag. e.g the code:

<meta name="viewport" content="initial-scale=2.0,width=device-width" />

scales the viewing area of the browser so it is twice the width of the device. It means the viewing area will be scaled so the it looks like the website is being viewed on the device that is twice the width of the actual device.

A simple way of changing the browser viewport is to simply resize the browser window.

Finding CPU usage using top command

Fri, 10 Feb 2017 00:00:00 GMT

The top command can be used in combination with bash tools to find CPU usage. The article: Finding CPU usage from top command describes how to find CPU usage using the top command.

The following command extracts CPU usage from the output of top command. The top command is run in batch mode so its output can be extracted:

cpu_usage=$(top -b -d1 -n1|grep -i "Cpu(s)"|head -c21|cut -d ' ' -f3|cut -d '%' -f1)

On some systems, in the above command f3 should be replaced with f2. The command returns the value for ‘us’, which is the percentage of the total CPU time spent running un-niced user processes. The output of the above command can be saved in a variable and sent by email using a command like mailx.

For example the following command will send an alert email to the system administrator using gmail server:

echo "Hello Admin. The CPU usage on server is $cpu_usage%. Please check!" > /tmp/mail_body.txt
mailx -s "CPU usage on server is more than 70%. Please check!." -r "reply_email" -S \
smtp="smtp.gmail.com:587" -S smtp-use-starttls -S smtp-auth=login -S \
smtp-auth-user="gmail_user" -S smtp-auth-password="gmail_password" -S ssl-verify=ignore recepiant_email &lt; /tmp/mail_body.txt
rm -rf /tmp/mail_body

Monitoring RAM usage on server

Fri, 10 Feb 2017 00:00:00 GMT

The RAM on a server can be monitored using /proc/meminfo and awk commands. /proc/ is a special file system created by the Linux kernel which contains kernel related information such as CPU and memory usage. /proc/meminfo contains information about memory usage.

The following bash script can be used to extract memory usage information from /proc/meminfo. It uses the awk text processing command for parsing memory information. It uses bc command for performing floating point operations. The bash scripting language does not support arithmetic operations on floating point numbers so an external tool like bc is needed

# the total memory. the MemTotal row is extracted using awk. the second column of this row is then extracted and saved to total variable
total=$( awk '/MemTotal/ {print $2}' &lt; /proc/meminfo )
# the available memory. the MemAvailable row is extracted using awk. the second column of this row is then extracted and saved to available variable
available=$( awk '/MemAvailable/ {print $2}' &lt; /proc/meminfo )
# the formula for calculating percentage of available memory is piped to bc command. the output of bc command is saved to memory_available variable
memory_available=$(echo "($available*100/$total)" | bc)
# if the available memory is less than 30%
if [ "$memory_available" -lt "30" ];
then
    # the body of the alert email is saved to a temprary file
    echo "Hello Admin. The Memory usage on server is $memory_available%. Please check!" > /tmp/mail_body.txt
    # the mailx command is used to sent the email. it sends the email using gmail server.
    # this is useful on servers that have limited access to email servers such as Google Compute instance
    mailx -s "Memory usage on server is more than 70%. Please check!." -r "reply_email" -S smtp="smtp.gmail.com:587" -S    \<br/>smtp-use-starttls -S smtp-auth=login -S smtp-auth-user="gmail_address" -S smtp-auth-password="gmail_password" -S \<br/>ssl-verify=ignore recepiant_email &lt; /tmp/mail_body.txt
    # the temprary file is removed
    rm -rf /tmp/mail_body
fi

Configuring Apache to process html files as php

Mon, 06 Feb 2017 00:00:00 GMT

To allow Apache web server, to process files ending with .html as PHP files, we need to do two things. First we need to set the content-type HTTP header for .html files to text/html. Secondly, we need to tell the Apache web server to process HTML files as PHP files.

Usually this can be achieved by adding following two lines in .htaccess:

AddHandler application/x-httpd-php .html
AddType text/html .html

The above code should work if PHP is configured as an Apache module, but may not work for other configurations like FastCGI.

To allow Apache to serve files ending with .html as PHP files, we need following code in vhost.conf file for the required virtual host:

<IfModule mod_fcgid.c>
    <Files tilde (\.html)>
        SetHandler fcgid-script
        FCGIWrapper /var/www/cgi-bin/cgi_wrapper/cgi_wrapper .html
        Options +ExecCGI
        allow from all
    </Files>
</IfModule>

The above code is for FastCGI configuration. It will differ for other configurations like mod_php. After editing the file we need to run following command for re-configuring the web server:

/usr/local/psa/admin/sbin/httpdmng --reconfigure-all

The article: How to process files with .html extension through PHP (Linux) describes how to process .html files through PHP on Linux.

Fetching PTR DNS records using Reverse DNS

Thu, 02 Feb 2017 00:00:00 GMT

Reverse DNS or RDNS is the process of fetching the host name for a given IP address. Reverse DNS works similarly to forward DNS. It is used in different situations. A common use of RDNS is to resolve an IP address to a host name. The process of resolving an IP address to a host name requires fetching the PTR record for the given IP address. It works as follows:

The DNS resolver sends a request for an IP address. for example 192.168.1.1. The DNS resolver reverses the IP address and appends the domain: .in-addr.arpa. So the request now becomes: 1.1.168.192.in-addr.arpa. The DNS resolver sends a request for the PTR record for: 1.1.168.192.in-addr.arpa to the root name server.

The root name server forwards the request to the DNS server that is in charge of Class A IP addresses which is the classification for IP addresses that start with 192. In most cases the request is sent to the DNS servers of a “Regional Internet Registry” for example ARIN for North America, APNIC for Asia Pacific and RIPE for Europe.

The “Regional Internet Registry” then forwards the PTR request to the DNS servers of the ISP that owns the IP address block. The ISP’s DNS server will forward the PTR request to the organizations DNS servers. The organization’s DNS servers should then respond with the contents of the PTR record.

The process of Reverse DNS lookup is similar to a normal forward DNS lookup in that each section of the name is looked up. For example first it looks up the DNS servers for 192.in-addr.arpa then for DNS server for 168.192.in-addr.arpa and then DNS servers for 1.1.168.192.in-addr.arpa.

PTR DNS records can be regarded as a pointer to a DNS record. It essentially points to a host name. Configuration of PTR records must be done by the ISP first. The ISP’s DNS server usually has a NS or nameserver record with name: 1.168.192.in-addr.arpa and value: ip-of-organizations-dns-server. The organization’s DNS server then adds a PTR record with name: 1 and value: required-host-name.

PTR records are commonly used to verify email servers. An email server with an invalid PTR record will not be able to send email since it is not trusted by most email servers.

What is reverse DNS and why should you care? is a useful article that describes in layman’s terms how Reverse DNS and PTR records work. It explains the importance of Reverse DNS and PTR records and how they are used by Web Analytics software and email clients.

Server Migration with Plesk

Thu, 02 Feb 2017 00:00:00 GMT

Introduction

Plesk is a commercial platform for managing website hosting. The cost of using Plesk depends on the number of domains to be managed, type of hosting (dedicated or VPS), addon products such as anti virus etc. Plesk allows managing FTP, Email, web site hosting, DNS and more. The user interface is easy to use and well organized. Plesk also has excellent online documentation, which makes it easy to troubleshoot problems.

This articles describes my experience with migrating server data from one Plesk instance to another using the Plesk Migrator Extension. I had followed the Plesk Migration Guide.

Installation and Pre-requisites

Log in as administrator to the Plesk destination server. Next go to “Tools and Settings”, then “Updates and Upgrades”, then go to Add/Remove components. After that select “Install” next to the Plesk Migrator component. Alternately, Plesk may be installed from the command line by running the plesk installer command.

In order to use the Plesk Migrator, certain TCP ports need to be opened on the source and destination servers. The installation of the Plesk Migrator is documented in this installation guide.

Plesk supports migration from following platforms: Plesk for Linux and Plesk for Windows, Cpanel, Confixx, Helm, Plesk Expand and Parallels Pro Control Panel for Linux.

Following data is transferred: service plans, subscriptions with associated domains and website content. Website content includes database, files and email. The settings of Plesk services such as Php, Firewall, Apache are not transferred.

Migrating from the web interface

Creating a migration task

Log in to Plesk as admin user. Then go to “Server Management”->“Extensions”->“Plesk Migrator”->“Start a New Migration”. For the Panel Type option select Plesk. For the ip address, enter the ip address of the Source Plesk server. For the root login field enter root. For the Authorization field, select either “User/password” or SSH keys.

If “User/password” option is selected then enter the password for the root user on the source server. If the SSH keys option was selected, then enter the path to the private key file used to login to the target Plesk server.

Next click on Prepare Migration. The Plesk Migrator will attempt to fetch information about the different subscriptions and domains. Plesk will now create an entry in the migration list. The migration entry can be edited any time.

Adding subscriptions

Click on the migration entry. This will open the add subscription tab. On this tab we can select the subscriptions to migration and the content to migrate. For example databases, email and website files. On this page, we must select the subscriptions to migrate. We cannot migrate individual domains within a subscription.

Migrating the subscription

After selecting the subscriptions to migrate, we need to click on Migrate. Plesk then runs pre-migration checks which displays the list of possible issues. After these issues have been corrected, click on Refresh. Once the pre-migration check returns a clean report, click on “Start migration” to being the migration. The migration progress can be viewed on the overview tab.

The migration process displays a status report for each subscription that is migrated. After a subscription has been migrated, we can re-sync the subscription by clicking on rsync besides the subscription. After the migration has completed, we can click on “Finish migration” to remove the migration from the list of on going migrations.

Assigning IP address to migrated subscription

When migrating a subscription, we will most likely want to change its IP address. We can do this from the subscriptions list tab. On this tab we need to select “Change IP mapping” from the “IP addresses” menu and then select the correct option.

Migrating from the command line

Running the pre-migration check

Copy a sample configuration file from /usr/local/psa/admin/plib/modules/panel-migrator/backend/conf/samples/ folder to /usr/local/psa/var/modules/panel-migrator/conf/. The available configuration files are: plesk-unix.simple.ini, cpanel.simple.ini, confixx.simple.ini, and ppcpl.simple.ini. I chose the plesk-unix.simple.ini file since I wanted to migrate from a Plesk server running on Linux. After copying the file, rename it to config.ini.

Next we need to edit the config.ini file and specify the ip addresses of the source and target servers and also the login information for both servers.

Generating the migration list

Next we need to generate the migration list by running the command: /usr/local/psa/admin/sbin/modules/panel-migrator/plesk-migrator generate-migration-list. This will create the file migration-list in the folder /usr/local/psa/var/modules/panel-migrator/sessions/migration-session/.

The migration list is a list of objects to migrate such as customer and re-seller account, service plans, domains etc. We can edit the migration list and remove domains that should not be migrated or re-assign them between customer accounts. This is one advantage of migrating from the command line. It allows us to select which domains to migrate and also makes it easier to troubleshoot errors.

Migrating the subscriptions

Next we need to determine which IP addresses should be used for the migrated subscriptions. The simplest method is to create a simple text file called ip-address-mappings which contains the source and target IP address on each line separated by space. We can use the —ip-mapping-file command line argument. After that we need to run the migration check using the command: /usr/local/psa/admin/sbin/modules/panel-migrator/plesk-migrator check. This will run a migration check. The results of the check will be saved to a file. The issues mentioned in the file should be corrected and the command should be repeated.

After that we need to run the command: /usr/local/psa/admin/sbin/modules/panel-migrator/plesk-migrator transfer-accounts —ip-mapping-file ip-address-mappings. This will start the migration. After the migration has completed, we can test the result of the migration with the command: /usr/local/psa/admin/sbin/modules/panel-migrator/plesk-migrator test-all.

Conclusion

Plesk is widely used for managing website hosting. It is easy to manage and has lots of features that greatly simplify web host management.

Rocket Chat installation tips

Thu, 12 Jan 2017 00:00:00 GMT

The following script can be used to run an instance of Rocket Chat:

#!/bin/sh

if [ $(ps -e -o uid,cmd | grep $USER | grep rocketchat | grep -v grep | wc -l | tr -s "\n") -eq 0 ]
then
        export PATH=/usr/local/bin:$PATH
        export MONGO_OPLOG_URL=mongodb://localhost:27017/local
        export ROOT_URL=http://your-domain/
        export MONGO_URL=mongodb://localhost:27017/your-database
        export PORT=1000
        export MAIL_URL=smtp://gmail_email_address:password@smtp.gmail.com:465
        /root/.nvm/versions/node/v4.5.0/bin/forever start --minUptime 1000 --spinSleepTime 10000 --sourceDir path-to-rocket-chat main.js -l /var/log/forever.log -o /var/log/forever-out.log -e /var/log/forever-err.log
fi

The above command checks if an instance of Rocket Chat is already running. If it is not running, then the Rocket Chat is run using the forever command. The forever command ensures that the Rocket Chat keeps running. It restarts the Rocket Chat automatically in case it stops.

The forever script can be installed with the command: npm install forever. The above script may be run when the system boots up using the following cron job command:

@reboot /root/scripts/rc_starter.sh > /dev/null 2>&1

If your Rocket Chat upgrade fails with the error:

Your database migration failed

Then you may need to unlock the migration in mongodb and also increase database version to the next version.

You can do this by accessing the mongo shell and then entering the rocketchat database using the command: use rocketchat. After that run the command:

db.migrations.update({_id: 'control'},{$set:{locked:false,version:19}})

Replace 19 with the current version of your database migration.

See the Rocket Chat Database Migration guide for more information on migrating Rocket Chat instances.

Designing APIs

Fri, 04 Nov 2016 00:00:00 GMT

Application Programming Interfaces (APIs) may be regarded as the building blocks of software applications. APIs provide a subset of the functionality of a system to an application. An API may be regarded as a set of libraries that allows an application to be used by multiple applications.

Since APIs can be used by multiple applications, it is import for APIs to be well written and concise. APIs should be carefully planned. The following points may be considered when designing APIs:

Authenticating

The API Your API should be called by valid users that are authorized and authenticated.

Caching API results

Your API should cache the results of API call. This will allow your API to handle requests more quickly, and it will be able to handle more requests per second. Memcache can be used to cache results of API call.

The API architecture

RESTFul APIs have less overhead as compared to SOAP based APIs. SOAP based APIs have better support for authentication. They are also better structured then RESTFul APIs.

API documentation

Your API should be well documented and easy for users to understand.

API scope

Your API should have a well defined scope. For example will it be used over the internet as a public API or will it be used as private API inside corporate intranet.

Device support

When designing your API you should keep in mind the devices that will consume your API. For example smart phones, desktop application, browser based application, server application etc

API output format

When designing your API you should keep in mind the format of the output. For example will the output contain user interface related data or just plain data. One popular approach is known as separation of concerns Separation of Concerns. For example separating the backend and frontend logic.

Rate limiting and throttling

Your API should implement rate limiting and throttling to prevent overuse and misuse of the API.

API versioning and backward compatibility

Your API should be carefully versioned. For example if you update your API, then the new version of your API should support API clients that use an older version of your API. Your API should continue to support the old API clients until all the API clients have migrated to the new version of your API.

API pricing and monitoring

The usage of your API should be monitored, so you know who is using your API and how it is being used. You may also charge users for using your API.

Metric for success

You should also decide which metric to use for measuring the success of your API. For example number of API calls per second or monitory earnings from your API. Activities involved in the development of your API, such as research, publication of articles, open source code, participation in online forums etc may also be considered when determining the success of your API.

API Response time

Your API should have a low response time. It should be possible to monitor the API response time and notify the developer if the API response time above average.

API Error Handling

In case of errors, your API should return a suitable message to the API client. It should also send notification email to the API developer.

Estimation of cost involved

You should also calculate the cost of developing and deploying your API. For example how much time it will take you to produce a usable version of your API. How much of your development time the API takes.

Updating your API

You should also decide how often to update your API. For example how often should new features be added. You should also keep in mind the backward compatibility of your API, so updating your API should not negatively affect your clients.

Deploying APIs on Google Cloud Endpoints

Thu, 03 Nov 2016 00:00:00 GMT

Introduction

Google Cloud Endpoint is a service that allows developing, deploying and managing APIs on Google Cloud. This article describes how to get started with Google Cloud Endpoint.

Cloud Endpoint is very well documented. Quick start guides are available which describe how to use Cloud Endpoint with different Cloud Backends such as App Engine, Compute Engine and Container Engine.

I had followed the Quickstart for Endpoints on Compute Engine. It describes how to deploy a sample API on a Compute instance. The sample API is based on Python language but can easily be substituted for other programming languages such as PHP or Java.

Deploying the sample API to Cloud Endpoint

Deploying a sample API to Cloud Endpoints requires Google Cloud SDK, a running Compute instance and a proxy server based on Nginx called Extensible Service Proxy (ESP). Deployment of the sample API requires several steps and is described below:

Configuring the pre-requisites

Create a new Cloud Platform Console Project or select an existing project. Note down the project id. Enable billing for this project
Install Curl. Then Download and install the Google Cloud SDK. These two steps should be done on both main server and your local development machine
Update the Cloud SDK, install the endpoint components and set the default project id to your [YOUR_PROJECT_ID]. Use following code:
```
gcloud components update
gcloud components install beta
gcloud config set project [YOUR_PROJECT_ID]
```
Clone the sample API from the source code repository with the following command:
```
git clone https://github.com/GoogleCloudPlatform/python-docs-samples
```
Change to the directory that contains the sample code. Use following command:
```
cd python-docs-samples/appengine/flexible/endpoints
```
In swagger.yaml file, replace all instances of YOUR_PROJECT_ID with your project id. Swagger is an open standard for specifying APIs. Swagger is now known as Open API Specification

Deploying the API

To deploy the API use following command: gcloud beta service-management deploy swagger.yaml. This command will return several lines including the server name and service version.

Create Compute instance

Next create a new Compute instance or use an existing instance. On the settings page for the instance configure the meta data. Add a new meta data item with key: endpoints-service-name and value: YOUR-PROJECT-ID.appspot.com. Next add another meta data item with key: endpoints-service-version and value same as the service version returned by the API deployment command.

Running the sample API

To run the sample API, on your local machine set the zone for your project with the command:

gcloud config set compute/zone [YOUR-INSTANCE-ZONE]

In the API directory python-docs-samples/appengine/flexible/endpoints, copy the API files from your local machine to your server. Use following command:

gcloud compute copy-files * [INSTANCE-NAME]:/full-path-to-sample-api-files

Next connect to your instance with the command:

gcloud compute ssh [INSTANCE-NAME]

Next set up a Python virtual environment on your main server using the commands:

sudo easy_install virtualenv
virtualenv

These commands should be run on your main server in the same folder that contains the API code.

Next install the requirements of the API sample using the command:

./bin/pip install -r requirements.txt

After that start the gunicorn web server with the command:

./bin/gunicorn -b :8081 main:app

This will start the API server on port 8081. Next we need to install and configure the Extensible Service Proxy (ESP).

Installing and configuring the Extensible Service Proxy (ESP)

The ESP is a proxy server based on Nginx. It sits in front of your API back-end and provides Authentication, API Key Management, Logging and other Endpoint Management features.

To install the ESP run the command:

sudo apt-get install endpoints-runtime

on your main server. If the package cannot be installed then you need to add the correct package repository to your package manager. See the quick start guide for more information.

Next in /etc/default/nginx set the port number to 80 and restart the ESP with the command:

service nginx restart

Sending requests to the sample API

To send requests to the API, create an API key on the API credentials page

Next make a HTTP request to the API using the command:

curl -d '{"message":"hello world"}' -H "content-type:application/json" http://[IP_ADDRESS]:80/echo?key=${ENDPOINTS_KEY}

Replace IP_ADDRESS with the external IP address of your instance. Replace ENDPOINTS_KEY with your API key.

Next view the activity graphs for your API in the endpoints page.

Updating the sample API

If you have made changes to your API, then you need to redeploy the API. To redeploy the API, make changes to your API source code and then edit the swagger.yaml file to reflect the changes. Then redeploy the API with the command:

gcloud beta service-management deploy swagger.yaml

Next the meta data values for your compute instance need to be updated with the correct API service version number.

If you have deleted your API by mistake like me, and you want to restore it, then you need to run the command:

gcurl -X DELETE https://servicemanagement.googleapis.com/v1/services/endpointsapis.appspot.com

gcurl is simply the curl command with authentication variables for Google Cloud. It is setup as a bash command. To setup gcurl we need to first follow the getting started guide for service management. The configuration of gcurl is described in the installation of oauth2l. I had run into this problem and had to post undelete a Google Service API on Stackoverflow.

Conclusion

Google Cloud Endpoints is a complete platform for developing, deploying and monitoring APIs. It allows deploying scalable, well managed APIs.

Developing APIs using Google Apps Script

Thu, 03 Nov 2016 00:00:00 GMT

Introduction

Google Apps Script is a platform for developing scripts for Google Products. It allows developers to interact with Google Products from JavaScript. It also allows extending the functionality of Google Products using JavaScript.

How it works

To use Google Apps Script you need to sign in to Google App Scripts with your Google Account. This will open a page that introduces Apps Script. Click on start scripting to continue to the script editor.

The script editor allows developers to manage their Google Apps scripts online. The editor has a useful auto complete feature that displays information on JavaScript classes.

Google Apps Script provides JavaScript classes for interacting with Google products such as Google Sheets. For example the following code returns a sheet object which represents the active Spreadsheet object in GoogleSheets:

var sheet = SpreadsheetApp.getActiveSheet();

The script editor allows running the JavaScript code from the editor window. It also allows deploying the Apps Script project as a standalone web app or as an Android add-on or as a Chrome Store Add-On. Google Apps Script also provides an API that allows executing an Apps Script from different environments such as .Net, PHP, Java etc.

The Google Apps Script documentation provides easy to follow 5 minute tutorials for learning Apps Script. I was able to easily follow this tutorial which allows creating a new Google Document and emailing the link.

Other uses of Apps Script include adding custom functions and user interface features to Google Apps products, interacting with external APIs and Google services and connecting with external databases such as MySQL.

Google Apps Script also supports triggers which are functions that are executed when certain events occur, such as opening a word document. We can easily create powerful APIs with Google Apps Script. These APIs can be consumed by different applications.

Conclusion

Google Apps Script is an excellent platform for interacting with Google Products such as Google Sheets, Google Docs, Google Sites etc. It can also be used to extend the functionality of Google Apps.

Deploying WSO2 Api Manager

Thu, 03 Nov 2016 00:00:00 GMT

Introduction

The WSO2 API Manager is a platform for managing APIs. It allows deploying, monitoring and testing APIs. It is a complete API management platform that allows building scalable APIs.

Installation

The WSO2 API Manager is well documented. Its installation is described in The WSO2 API Quick Start Guide. The WSO2 API Manager can be installed in three simple steps:

First we need to install Oracle Java JDK version 1.7 or 1.8. Then we need to set the JAVA_HOME variable. After that we need to download the WSO2 API Manager. Next we need to start the WSO2 API Manager with the command:

{APIM_HOME}/bin/wso2server.sh

After that we should be able to access the WSO2 API Publisher on https://hostname:9443/publisher. We can login with user admin and password admin.

Using the WSO2 API Manager

The WSO2 API Manager consists of an API Publisher and an API Store. The API publisher allows developers to publish APIs. The API store allows developers to use or subscribe to APIs.

To publish an API you have to login to the WSO2 API Publisher on https://hostname:9443/publisher with user name admin and password admin. After logging in click on the Deploy Sample API button. It deploys a sample API called PizzaShackAPI into the API Manager. The API will now show under the APIS tab.

To subscribe to the API login to the API store at: https://hostname:9443/store with user name admin and password admin. The newly created API is visible under APIs menu. Click the API to open its menu.

Next select a default application and an available tier and then click on Subscribe. Next click on the Production Keys tab and then click on generate keys to generate an access token to invoke the API. You have successfully subscribed to the API.

Next go back to the APIs menu and click on the PizzaShackAPI. Next click on the API console tab. Next expand the GET method and then click on the Try it out button. The response of the API invocation shows a list of menu items.

Conclusion

APIs play an important role in Software Development. Web Applications in particular benefit greatly from APIs. A well built API can be consumed by applications on different platforms and can form the backbone of a multi platform application.

The WSO2 API Manager is an excellent platform for managing APIs. It is a scalable platform with useful features for managing different aspects of APIs.

Samba - An open source File Server and Active Directory Controller

Thu, 03 Nov 2016 00:00:00 GMT

Introduction

Recently I had the task of setting up central user management for Windows servers. This blog post describes my experience with trying to setup central user management for Windows servers in an open source Linux based environment.

My client is fond of free and open source tools, so I had to find a free and open source tool that provides central user management for Windows based servers. After some searching on the internet, I came across Samba.

Samba is basically a file sharing server that can be used by all devices that support SMB/CIFS. These include Linux and Windows based servers. Version 4 of Samba, which was released in 2012 had a new feature that allowed it to function as an Active Directory Domain Controller.

This is a very useful feature as it allows Windows based hosts to authenticate against Linux based Samba Authentication Server. This opens up the possibility of cross platform user management, which is very useful in a heterogeneous I.T environment with different operating systems.

Installation and configuration

Configuring Samba to work as an Active Directory Domain Controller is simple. Installing and configuring the Samba server and configuring a Windows server to authenticate against the Samba server takes less than 20 minutes.

Instructions on configuring Active Directory on Samba are given on the Samba Wiki. Instructions on connecting a Windows client to a Samba AD DC are also given. Of course if you have specific requirements then you can read the articles on the Samba Wiki or read the Samba mailing archives. To install Samba and configure it as a Domain Controller follow these steps:

Install Samba

I used the samba package repository provided by SerNet. SerNet provides package repositories for the major Linux distributions. Create an account on SAMBA+. Then download and install the package repository and package signing key. Follow the given instructions.

Then install the Samba packages. For Centos enter the following command:

yum install -y sernet-samba sernet-samba-ad sernet-samba-client

Enter following commands to verify correct installation. Samba -V and smbclient -V. These commands return the version of Samba server and Samba client that was installed. Its best to install the latest version.

Provisioning The Samba Active Directory

This configures Active Directory Domain Controller. It also configures Kerberos on the server. Kerberos is a mechanism for encrypting authentication information. It is used by Active Directory Domain Controller to provide secure authentication to the domain clients. The Kerberos website has useful information on Kerberos configuration.

To start the Samba AD DC provisioning enter the following command:
```
samba-tool domain provision --use-rfc2307 --interactive
```
You will then be asked about the following:
- Samba Realm (Set this to a sub domain of your main domain. e.g WIN.EXAMPLE.COM)
- Domain (Set this to the first part of your Samba Realm. e.g WIN)
- Server Role (Set this to dc since we are trying to set up a domain controller)
- DNS Backend (Set this to SAMBA_INTERNAL)
- DNS forwarder IP address (Set this to NONE or the IP address of your DNS forwarder)
- Administrator password (Set this to the admin password for your domain controller)
If all goes well the command will end without errors. You can read more about the Provisioning command on “Setup a Samba Active Directory Domain Controller”.
Start samba.

Setup a startup script for Samba. You can use the Samba4/InitScript. Start Samba. e.g on Centos enter the command:
```
systemctl start samba-ad-dc
```
Test Samba Domain Controller

Enter the command:
```
smbclient -L localhost -U%
```
It should return some information about your Samba configuration. Enter the command:
```
smbclient //localhost/netlogon -UAdministrator -c 'ls'
```
This command will connect you to the netlogon share, using the Domain Administrator account, created during provisioning. If these commands fail then check the Samba AD DC Troubleshooting guide.
Configure DNS configuration file

Add the IP address of your DNS server. This DNS server must have the correct DNS records created. Following DNS records need to be created for your DNS zone. Your DNS zone is same as the Realm name you gave during provisioning. Login to your DNS server and create the following records for your DNS zone:

Create a record with the name _ldap._tcp and type SRV. use priority 0, weight 100, port 389 and the name of your Samba server.
Create a record with the name _kerberos._udp and type SRV. use priority 0, weight 100, port 88 and the name of your Samba server.
Create a record with the host name of your Samba server and type A. Use the ip address of your Samba server.
Create a record with the name _ldap._tcp.dc._msdcs and type SRV. use priority 0, weight 100, port 389 and the name of your Samba server.
Verify DNS configuration.

Enter the following commands on your Samba server to verify correct working of the DNS. If you receive any errors, check your system log.

host -t SRV _ldap._tcp.WIN.EXAMPLE.COM.
host -t SRV _kerberos._udp.WIN.EXAMPLE.COM.
host -t A addc.WIN.EXAMPLE.COM.
host -t A WIN.EXAMPLE.COM.

Kerberos Configuration.

Kerberos configuration can be quite complex. Thankfully the Provisioning tool takes care of the Kerberos configuration. During provisioning the file**/usr/local/samba/private/krb5.conf** is generated. Location of this file can vary depending on your system. Copy the file krb5.conf to /etc/krb5.conf. This is the main kerberos configuration file. You can read more about configuring Kerberos on the Kerberos website.

Testing Kerberos

Obtain a Kerberos ticket by using the kinit command. Enter following: kinit administrator@WIN.EXAMPLE.COM. You will then have to enter the administrator password that you gave during provisioning. To verify that the command worked enter the command: klist. It should return the Kerberos ticket that you just created.

Time Synchronization

Use NTP (Network time protocol) on your Samba server and Windows clients. This ensures that the time on the Server and Clients is synchronized. This is very important for correct working of Kerberos.

Samba does not have a full fledged forum but instead uses a mailing list. You can ask a question by posting to a mailing list. I was interested to know if its possible to synchronize FreeIPA users with Samba AD DC users. I asked on their general samba mailing list and also mentioned my experience with installing Samba.

Conclusion

Samba is an excellent open source tool that offers the possibility of managing Linux and Windows users from a single tool. Samba is usually used as a file sharing server in enterprises and is a useful tool to learn.

References:

FreeIPA - An open source centralized user management system

Mon, 24 Oct 2016 00:00:00 GMT

Introduction

This blog post is about FreeIPA, a very useful open source centralized user management solution from Red Hat. I had the opportunity of installing FreeIPA in a virtualized environment. I want to share my experience in this blog post.

FreeIPA can be considered as the Linux equivalent of Active Directory. IPA stands for Identity, Policy and Audit. FreeIPA is actually a lot more than a centralized user management system. Along with user management it also provides integrated DNS, Certificate Authority and Access Control.

FreeIPA is a collection of the following open source tools: A Linux (Fedora), 389 Directory Server (LDAP), MIT Kerberos, NTP, DNS, Dogtag (Certificate System). It also supports Active Directory sync. FreeIPA uses LDAP for storing its data which includes DNS, host and user information.

Why use FreeIPA

Centralized user management

Why would you want to use FreeIPA?. Well if you have hundreds of Linux based servers, for example in a virtualized environment, it would be really useful to manage users for all your servers from one place. FeeIPA lets you do that. FreeIPA provides a web interface and command line tools to manage users all from one place. Since FreeIPA uses LDAP for user management you can connect it to any LDAP client. Most open source tools support LDAP.

Quick and easy installation

Installation of FreeIPA has to be done once on the server and once for each client to be managed. Both client and server installations are easy and take less than 15 minutes. In less than 15 minutes you can have a centralized user management solution for free !. And it comes with a nice GUI as well !. Of course customizing and learning about FreeIPA will take some time. Red Hat has a useful guide titled “Linux Domain Identity, Authentication, and Policy Guide”, that explains all aspects of FreeIPA including installation and administration.

Installation and Configuring

Installing FreeIPA Server

Red Hat derived operating systems, e.g Centos and Fedora have FreeIPA in the default package repositories. Installing FreeIPA is as simple as entering the command :

yum install ipa-client

on Centos. The following command should be entered on Fedora

yum install freeipa-client

To install FreeIPA on other Linux systems you will have to install from source.

Configuring FreeIPA server

After installation the command:

ipa-server-install

has to be run. This starts the FreeIPA installer, which configures the FreeIPA. The configuration can be run interactively or silently by passing command line options. Installing an IDM Server, describes the complete configuration steps.

Installing FreeIPA client

To install the FreeIPA client on Centos you have to run:

yum install ipa-client

On Fedora run:

yum install freeipa-client

On other Linux operating systems you will have to follow the manual installation steps given in the Red Hat documentation.

Configuring FreeIPA client

After installation run the command:

ipa-client-install

to start the configuration. The client side of FreeIPA is based on SSSD Linux Daemon. SSSD is a caching daemon that sits between services such as Linux authentication service and the authentication storage. SSSD intercepts the request sent by the service and can return a cached response or it can send the request to another service such as a FreeIPA server. See Setting up systems as IDM clients for complete information on installing the FreeIPA client.

Conclusion

Once the client installation is complete, you can manage all your users, DNS, hosts, certificates and more from the FreeIPA web interface. FreeIPA is a great tool that combines several well known Linux solutions into an integrated identity management platform. It is very easy to install and configure. You don’t have to know the workings of the tools that are internally used by FreeIPA such as bind, open LDAP, DogTag. But learning about these tools will give you a better idea of how the FreeIPA works.

Installing FreeIPA client on Debian

Thu, 25 Aug 2016 00:00:00 GMT

Introduction

Recently I had the opportunity of installing FreeIPA client on servers running the Debian operating system. I will share my experience in this blog post. FreeIPA is an excellent centralized user management system from FreeIPA.

It is free and open source. FreeIPA has lot of features other than user management such as DNS, Certificate management, Encrypted Authentication and more. In my earlier blog post titled FreeIPA - An open source centralized user management system, I have described the installation and configuration of FreeIPA server

Installing FreeIPA Client

FreeIPA server and client are very easy to install on Centos and Fedora. Configuring a Linux system as an IDM Client, is a useful guide from RedHat on installing FreeIPA clients on RedHat based systems.

Installing FreeIPA on other operating systems is a lot more difficult. Even though RedHat provides a guide titled Manually configuring a Linux client, I was not able to get it to work on Debian. The HowTo guides on the FreeIPA wiki describes the process of installing FreeIPA on non Red Hat based operating systems and integration with different applications. However these guides were not useful.

Ubuntu provides FreeIPA server and client packages on Launchpad. Ubuntu is based on Debian, but is not Debian so I went on searching for a solution for Debian. I came across a thread titled “freeipa-client on Debian Wheezy” on the Ubuntu FreeIPA mailing list. It was posted by one of the contributors of the Ubuntu FreeIPA project. He created a package for Debian and provided a repository for easy installation.

Here is how to install FreeIPA client on Debian.

# First add the following 2 lines to /etc/apt/sources.list file:
deb http://apt.numeezy.fr wheezy main
deb-src http://apt.numeezy.fr wheezy main

# Then install the package signature verification key using:
wget -qO - http://apt.numeezy.fr/numeezy.asc | apt-key add -

# Update your package lists:
apt-get update

# Set your server host name to a fully qualified domain name. e.g server.domain.com
hostname server.domain.com
vi /etc/hostname
# Install FreeIPA client
apt-get install -y freeipa-client

# Create folder for storing certificate database
mkdir -p /etc/pki/nssdb

# Create empty database for storing certificates
certutil -N -d /etc/pki/nssdb

# Create folder
mkdir -p /var/run/ipa

# Remove existing FreeIPA client configuration
rm -f /etc/ipa/default.conf

# Run the script for installing FreeIPA client
ipa-client-install --no-ntp --no-dns-sshfp --mkhomedir

# Enable auto creation of LDAP user folders
echo 'session required pam_mkhomedir.so' >> /etc/pam.d/common-session

# Add following lines to /etc/nsswitch.conf file or update existing lines
passwd: files sss
group: files sss
shadow: files sss
# Reboot you server
reboot

Conclusion

After that you should be able to login to your server as a LDAP user. The user needs to be created on your FreeIPA server. I was able to get FreeIPA client working on Debian running Proxmox. In case you run into problems you an check out the bug list on Ubuntu FreeIPA forums

Proxmox - An open source Virtualization Platform

Thu, 25 Aug 2016 00:00:00 GMT

Introduction

This blog post is about my experience with using Proxmox Virtualization Platform. Proxmox is an excellent open source virtualization platform from the German company Proxmox Server Solutions GmbH. It is used by tens of thousands of users.

Proxmox Features

Centralized Web Interface

You have to first install Proxmox from an ISO file that you can download from the Proxmox website. Proxmox needs to be installed on each server that needs to run virtual machines. Proxmox allows you to manage all your virtual machines and hosts from a single web interface.

Proxmox is a very good option for building your own private cloud. Proxmox supports KVM and OpenVZ virtual machines. The current Proxmox version which is 4.0 no longer supports OpenVZ virtual machines. Instead it supports LXC containers.

Support for KVM virtual machines

KVM virtual machines consume fixed CPU and Memory and can run any operating system that supports virtualization, including FreeBSD, Linux, Windows etc. KVM virtual machines can run network gateways, NAS servers, monitoring servers. Basically any server that can be installed from ISO.

You can have all your mail servers, DNS, database, NAS servers etc all running as virtual machines. Installing a server is as simple as uploading the server ISO file to your Proxmox server and following the installation instructions

Support for OpenVZ virtual machines

OpenVZ virtual machines are light weight containers that share the host servers hardware resources. They only support Linux operating system. The latest version of Proxmox no longer supports OpenVZ. Instead it supports a similar container technology called LXC which stands for Linux Containers.

Easy Administration

Unlike other cloud platforms such as AWS and OpenStack, Proxmox is very easy to use and requires no expert knowledge. Creating virtual machines is as simple as clicking a button and filling out a short form.

The Proxmox user interface is very simple and easy to use. It does have complex features such as distributed storage, user privileges and backups, but the complex features are well hidden. To see resource usage statistics such as CPU and Memory of a virtual machine, all you have to do is click on a virtual machine and then click on the summary tab.

Changing RAM and Memory is as simple as clicking on the hardware tab and editing text fields. You can also easily take backups of virtual machines. Proxmox has fine grained user permissions that allow you to configure the actions that users are allowed to take. For example, permissions to start and stop virtual machines.

Conclusion

Proxmox is a very flexible and easy to use virtualization solution. It is an excellent option for hosting a private cloud. By simplifying server administration, Proxmox makes it easy to publish innovative services on the internet at low cost. Virtualization removes the need for large number of expensive servers in institutions and companies. It forms the backbone of cloud services.

Cloudinary CDN - A free Content Delivery Network

Thu, 18 Aug 2016 00:00:00 GMT

Introduction

Recently I had the opportunity of integrating a free Content Delivery Network (CDN) called Cloudinary with my clients website. I will describe my experience in this blog post. Cloudinary is an excellent Content Delivery Network that can be used to serve static website content such as images, video, audio, HTML, CSS, JavaScript files and more. Cloudinary has an excellent free plan with great features.

Actually Cloudinary supports any type of static content. If you need your website’s static content delivered with high reliability and good performance, then Cloudinary is a good option.

Why use Cloudinary

There are a few free CDN services such as Boostrap CDN, CloudFlare, Coral Content Distribution Network and Incapsula. However none of these Free CDN services could meet my requirements. I needed an easy to use Free CDN service that was reliable and had good performance. Only Cloudinary CDN could meet these requirements.

The main issue with other CDNs was that they offered too many services such as web hosting, web application firewall, DNS services etc. These services are usually integrated with their free CDN service and in most cases are offered for free but on a trial basis.

Features of Cloudinary CDN

Free plan

The features of Cloudinary CDN that I found most attractive are that its CDN service is free without any trial limits. Their free plan allows 7,500 images, 75,000 total images, 2G storage and 5G monthly viewing bandwidth. These limits can be doubled if you share their profile on social networks such as Twitter and Facebook.

Support for different file types

I needed a CDN that supported audio files for my WordPress plugin. My plugin allows users to listen to audio translations of Holy Quran verses, so I needed a CDN that supported MP3 files. Cloudinary provided the ideal solution.

Flexible Application Programming Interface (API)

One problem I faced when using the Cloudinary CDN was that when we upload media files to the CDN they get renamed automatically. Fortunately Cloudinary has an excellent API with many functions one of which allows us to rename uploaded files. I was able to easily use their API and rename the uploaded images.

While running some performance tests on the website I noticed the images take a long time to get served. I needed a free CDN that was easy to use with WordPress. Cloudinary again provided the solution.

Cloudinary has a plugin for WordPress called Cloudinary - Image management and manipulation in the cloud + CDN that allows easy integration with their Free CDN service and WordPress based websites. Once you have installed the plugin you can easily serve your blog images from the Cloudinary CDN.

Conclusion

Cloudinary is an excellent Content Delivery Network that has many features including image manipulation, free cloud storage, support for all file types, fast delivery, easy to use administration interface, Software Developer Kits (SDKs) in multiple languages and support for security and privacy. It has over 50,000 customers. In my opinion Cloudinary is a very useful cloud service that helps in the creation of innovative services.