Learning programming

    Programming is an interesting world where you apply your skills to build a software program that comes into life when it is run on a computer device.  The live software programs behave the way you have created them.  You can modify the program to change its behavior or you destroy it and rebuild a whole new program.  It gives you sense of pride for creating something and then maintain it.

    You can choose either classroom training or self learning using the learning material and resources available.  The classroom training speeds up your learning as the teacher will assist you when you are stuck.  However don't be sad if you can't take up classroom training, there are so many videos and reading materials on the web which will help you to self-learn.

Prepare yourself

    To learn programming, first and foremost thing that is needed is your interest to learn programming.  Interest is needed to learn anything, be it programming or something else.  It is indeed the most important need as you can acquire all other required skills as long as you have a strong interest to learn something.

“Live as if you were to die tomorrow. Learn as if you were to live forever.” - Mahatma Gandhi.

    In the early stage of learning, you may find it difficult if you are trying to write a new software program all by yourself.  You will make mistakes and the program will fail to run or it runs with the wrong results.  However by exploring yourself to write a program and troubleshooting it by going through each line of code will help you to develop a fine eye for details.  These learnings will take you a long way in writing bigger software programs.

“Anyone who has never made a mistake has never tried anything new.” - Albert Einstein.

    As you are learning to write complex software programs you will most likely get stuck at some point in time.  It is very important that you have lot of patience while trying to find the reason as why the software program is not working as expected or how to make progress while finding a solution.  Breakdown the program and problem statement into smaller units to speed up your investigation for narrowing down to the problem area.  Consider all possible scenarios including the external programs that your program depends on.  Your patience and persistence to make it work will definitely pay you off with success, so always believe in yourself.

“Believe you can and you are halfway there.” - Theodore Roosevelt.

    While writing the software program consider its full lifecycle.  i.e., consider how the program will be installed, behavior of the program during start and stop phases, how the request will be processed, what is the format of the input data etc.  To support the same you need to code the software program with many logical statements and sequence of instructions.  Don't hesitate to try to solve the problem in different ways, and validate each method to find a more optimal, readable and high performant solution.  This helps you to develop problem solving and analytical skills, and it helps you produce better programs in future.

    To create something totally new, it requires your creativity and a focused effort.  Using the programming language elements you can write a software program in different ways to achieve the same result.  With this flexibility you can be as creative as you can.  Be creative in building a complex software program by breaking it into smaller program modules and stack them up.  Designing smaller program modules as independent and having single responsibility will help you to reuse them in many different place.  Writing a totally new and complex program from nothing looks very difficult at the beginning, but once you are successful you will enjoy it.

    Lastly learn to visualize the whole software program, start from the smaller components and then to larger software programs.  Visualize their structure, their behavior and their interaction with each other or with external programs, their access points and security etc.  Visualizing the software program helps you to design a better software that is more modular, more optimal, more performant solution.

“It always seems impossible until its done.” - Nelson Mandela.

In summary, you need

• passion to learn
• problem solving and analytical skills
• visualize and exploring things
• lots of patience and never give up attitude

Prepare environment

    To learn programming, you need to prepare the environment to learn.  If you are going through classroom based trainings, not the virtual classroom, then most likely you already have what is required to learn.  If you are going to self learn, below items guide you in preparing your environment for learning.

    You need a device on which you can write a program and test.  i.e, you need a computer system which has a display and a keyboard.  It will be useful if it is connected to internet otherwise its not mandatory.  For example, it could be a laptop or a desktop that has any of Windows or Linux or Mac OS etc installed on it.

    Learning programming is nothing but learning to code in a specific programming language.  So you choose which programming language you plan to learn.  There are many programming languages and the list can be found easily, however in my opinion the basic concepts and the basic constructs remains almost the same.  Some of the popular ones are java, python, c, java script etc.

    Based on the programming language you choose you need to prepare a development environment on your computer system by installing it with the tools and resources that help you to write and test the software program.  For example, you need a text editor to write the software program, a language specific compiler program if there is any to translate the written program to executable binary and the run time environment to execute the binary.

    The runtime environment varies based on the programming language. For example languages such as C use the operating system as a runtime environment, where as languages such as java require java runtime environment installed on the operating system, and the javascript code is run on the web browsers etc.

    Writing the software program on a simple text editor could be difficult, so there are also IDE (Integrated Development Environment) tools that are available and they will help you in programming easily.  These tools integrates with compilers and run time environments and have cool features such as an ability to validate the code you write before you compile or run, assist you in coding by providing suggestions, debug or run the program in the IDE itself, integrates with version control etc.

    For an advanced learning or developing software you need additional tools such tools for  design, test, documentation, build, package, version control, performance analysis, simulators etc.  However for a beginner none of these tools are necessary.

    In summary, based on the programming language you choose to learn visit the programming language website, understand the system requirement and prepare the environment on your computer system accordingly.

    Or nowadays these tools and resources are also available on the web, so that you don't need to install anything on your system.  These web tools allow you to write the program using programming language of your choice and has capability to test them on the web page itself.  Behind the scene, the software program you write is generally run in a sandbox in the cloud and returns the result to you on the web page.  In my opinion these web tools are not mature yet to provide you all the options and experience to learn and develop, so it is best to prepare your local environment and learn.

Hope this was useful and wish you all the best for your learning journey!

Disclaimer:  I hope this information helps someone.  Above writing reflects my view based on my experience and learnings.  Please comment if you find any discrepancy.

Software program and programming

    Software programs have been there for many years now, and have advanced a lot in terms of technology, programming, designs and practices.  There are a large amount of population who are interested in learning programming and some find it difficult.  I hope programming related posts in my blog are simple to understand and are helpful in their journey of learning.

“A software program is a set of coded instructions that can be run on a computer to perform the specified operations.”

Fun software program

    Here the computer is any intelligent hardware device that reads the provided instructions and performs the requested operation.  For example, it can be a desktop computer, a laptop, a mobile phone, a tablet, an iot device, or an embedded computer processor, or a virtual hardware in the cloud, etc.

    As the technology is advancing everyday, many of the devices that we use everyday are becoming smart by embedding a computer and a software program.  For example smart TV, smart refrigerators, smart cars etc.

    The operations performed by a software program could be a compute operation on a given data, or to manage data, or to route the data over network, or to control the hardware and resources etc.  A software program is written to solve a user/business requirement and the instructions coded in the software program are aligned to perform operations as required for a given user/business needs.

    For example, an ecommerce software program has the set of instructions having logic to process the submitted shopping order request.  It takes the user order data, computes the total price of the products purchased, applies discounts, processes the payment and generates order confirmation.

  There are too many software programs in this world. There are software programs for providing ecommerce services, for generating financial report of a company, or for schools to manage the students academic records, or for game enthusiasts, or for connecting your mobile phone with your television, or for providing infrastructure service on the cloud, etc, etc, etc.

“A software program is called as a software product, if it has well defined user interface, documentation and support on how to use and workflows.”

    The software product is also interchangeably called as software or software application or simply an app.  Based on its use, different terms are used such as system software, application software, enterprise software etc.  Few examples are, Google Pay, Microsoft Word, Angry Birds game, Amazon Alexa, etc.

    More than one software that have similar user interface and integrates with each other are packaged into one bundle called Software Suite. Software Suite is generally designed in such a way that its collection of software provides a holistic experience to the user with different services offered together.  Few examples are Microsoft Office 365, Adobe Creative Cloud, VMware vCloud suite, Apple macOS etc.

    In my opinion a software program is more of a technical term, and I would use the term software program mostly in the technical documentation.

“The process of writing a sequence of coded instructions to prepare a software program is called as software programming or simply programming or coding.”

    To write a software program, you need to understand the user requirement, define the program design and translate the requirements into sequence of instructions with some logic.  You will then test written software program to verify whether the results are satisfactory, and this process is called as software testing.  There are a lot of programming practices and styles followed in the industry.

“The instructions in a software program are written in a formal language called programming language.”

    There are many programming languages with various levels of abstraction.

    A software program that will be run on a computer should have instructions in the language that the computer understands.  This is called machine language or machine code and these are binary codes. It is impractical for anyone to write big software programs in machine language.

    The assembly language is symbolic representation of machine code instructions that is human readable.  An assembler software translates the assembly language program into the machine code format that the computer understands.  These are low level languages and are very much tied to instruction set of the computer.  For example, 8051, 8086, x86, x64 assembly languages.  It is still impractical to write big software programs in low level languages. 

    To simplify writing a software program, the high level languages were developed.  The high level languages provide higher level of abstraction from machine language and uses natural language elements to simplify programming.  The software program written in high level language is translated to machine language using compiler software programs. You would then run the compiled software programs on the computer.  There are many high level languages and each have different features and styles for writing a software program.  For example, C, C++, Java, Python, C#, etc.

Software programs in different languages

    The software program that is written in any of above language is called as source code and the compiled code that runs on the computer is called as binary executable or simply binary.  The binary executable is then packaged and released to users along with documentation.  The software development methods and practices is a very vast topic to be included here.

Disclaimer:  I hope this information helps someone.  Above writing reflects my view based on my experience and learnings.  Please comment if you find any discrepancy.

Thread dump in Tomcat

In a multithreaded environment if the implementation is not perfect, then deadlocks are common scenarios. We can analyse the deadlocks by placing log statement at proper locations in the code and analyzing the log, however to exactly pinpoint a deadlock thread dump would be required. Using the thread dump, we can exactly point out what are the two threads which are fighting for the monitor and prints the stack trace of each thread. With the stack trace we can easily find the use case where the deadlock occurs. There are many ways of getting the thread dump and few of them are listed below.

Unix:
Thread dump can be obtained in the appropriate log by sending the command
kill -3 PID, where PID is the process id of the Tomcat jvm.

Windows:
If your tomcat is running as a windows console, then you can open the console and type <ctrl>+<break>. This would print out the thread dump on the console. Make sure you increase the screen buffer memory size to capture the entire thread dump. However If your tomcat is running as a windows service, then you can obtain the thread dump by using the Tomcat monitor.

More on Tomcat as Windows service:
The Tomcat service monitor when started will bring an icon on the system tray. Right click on the icon and select "Thread Dump" in the menu options. This will trigger the generation of thread dump and the thread dump stack trace will be logged to the file system based on the Tomcat service configuration.

With the default settings, the tomcat log redirection would be set to auto. This can be verified by opening the Logging tab of the Tomcat service editor. With the default configuration, the threaddump would be either in jakarta_service, stdout or stderr logs. Or if you have configured tomcat service to redirect to a specific file then you will find the stack trace in the configured file. However if you not configured the logging properly for the tomcat service, then the stack trace would be lost. You will have to configure the service with proper values for logging and restart the tomcat service.

There are also other hacks of sending the <ctrl>+<break> signal to the tomcat service. For instance there is an application called SendSignal.exe which can send <ctrl>+<break> signal to tomcat service by running the command: sendsignal PID, where PID is the process id of the tomcat service. The thread dump stack trace would be logged to file system based the tomcat service configuration. However it may not work in all scenarios based on the security settings. Atleast I couldn't get it working as it always gave me the error related to insufficient privileges, though both the service and sendsignal were run by the same user.

NOTE: sc queryex <service_name> will give you the process id of the service.

Tomcat as a Windows service

Tomcat, a very familiar name in J2ee world. An excellent example of how open source is growing so powerful in the server domain.

In Windows, Tomcat can be setup in two ways

1. Install the Tomcat as a standalone server

2. Install the Tomcat as a windows service

The first method is fairly simple, which is nothing but just extracting the tomcat archive and running the startup script to run the Tomcat. Normally in a developers local system he would just extract the archive and run the Tomcat, however for running the Tomcat on the server the second method is preferred. Again there are two ways of installing Tomcat as a Windows service.

1. If you are running the Tomcat installable exe file then follow the instructions to install the Tomcat as a service.

2. If you have just extracted the Tomcat and wish to setup Tomcat as a Windows service or If you have already installed the Tomcat with the default name and would like to install another Tomcat instance with different Windows service name, then there are set of steps one should follow. Below are more details of the same. Lets take a look of it one by one.

Under the Tomcat bin directory you would find two interesting executable files. Assuming that you are using Tomcat with 6.x version, however if you are using different version then the file names would be different accordingly.

Tomcat6.exe - This is a service application for running Tomcat6 as Windows service

Tomcat6w.exe - GUI application for monitoring and configuring Tomcat services.

By invoking these executables with specific parameters we can install, update, edit, run, stop, monitor, delete the Windows service. However as these parameters are more techincal, Tomcat also provides a batch script, service.bat, with two simple commands install/remove. The usage of this batch script is as below.

Usage: service install/remove [service_name]

The service_name is the name of the Windows service and is an optional parameter. If you do not specify the service name then by default it will attempt to install/uninstall the service with name tomcat6.

If you are planning to write a batch script of your own, similar to service.bat, then refer to Windows service HOW-TO in locally installed tomcat documentation or in the internet with link http://tomcat.apache.org/tomcat-6.0-doc/windows-service-howto.html. It would also help you by looking at what all are being configured in the service.bat script provided by Tomcat.

Tomcat6.exe is used to install, edit, run, stop and delete the Tomcat service. However if you are looking for a GUI interface for editing the service configuration then the answer to this would be Tomcat6w.exe. Tomcat6w.exe is used only for configuring and monitoring the Tomcat services, so it is a prerequisite to have the Tomcat service already installed for running Tomcat6w.exe. There are two modes of running this application.

1. Run as a GUI based service editor

2. Run as a GUI based service monitor

By just running this exe without any parameter it would by default attempt to open the service editor with service name as file name without 'w'. Here it would be Tomcat6, assuming that you are running Tomcat6w.exe. If your service name is different, i.e., other than the Tomcat6, then you can open this service configuration by running the command: Tomcat6w //ES//<service_name>. Or much simpler solution is to make a copy of this executable file and rename it as <service_name>w.exe. In this case by just running this new file it would open the editor for windows service with the name as <service_name>.

To run the Tomcat as a monitor, you need to specify the command as: Tomcat6w //MS//<service_name>. It would be beneficial if you write a simple batch script, say monitor.bat, under Tomcat bin directory with just above command for monitoring purpose. Again remember if you have made a copy of Tomcat6w with service name as said above then the command can also be <service_name>w //MS//

In summary, below steps help you in managing the Tomcat as a windows service.

1. Install: In command prompt, cd to Tomcat bin and run the command: service install <service_name>

2. Edit: Make a copy of file tomcat6w.exe and rename it to <service_name>w.exe. Just run this file to open service editor.

3. Monitor: Prepare a batch script with command: tomcat6w //MS//<service_name> (or) <service_name>w //MS//. Run this script to open service monitor.

4. Uninstall: In command prompt, cd to Tomcat bin and run the command: service remove <service_name>

Note: The Windows service name and service display name are different and service name is case insensitive.

SoapUI for Data Setup - Use Case

With Soap UI having very useful features, I find this tool helpful for more and more purpose than just testing and coverage. In one of the product implementation, we were trying to setup the initial necessary data (basic and test data) in easy and best way. We needed the initial setup of few organization profiles and users to proceed further in the product implementation. We had three options - 1. Create manually through UI (painful), 2. Create through web service, 3. Write stored procedures (complicated). As second option seemed easy and clean we built the web service requests and ran each of them to setup data. However we felt there was still some more scope for improvement on this. And then flashed the test suite functionality of SoapUI!!

Once I had the idea it was very easy to setup the project on SoapUI with one test case under a test suite. That's it. I created one test step (web service request) for each profile to be created. However user creation required some of the profile information. Property Transfer came in handy to transfer the required values from the response of a specific profile creation request to the user creation request. (I tried many ways to do this with Property Expansion, however couldn't find a way to do this).


The snapshot of the setup is as shown in the figure. As you can see there are multiple SOAP requests, one each for profile/user creation. The property transfer worked fine, however I observed one weird behavior. Though the response xml did not contain any namespace prefix, In Property Transfer I still had to declare the namespace and mention the xpath elements with this namespace prefix. I still need to find a reason for this behavior.

Another notable thing to mention here is to turn on the option to close http connection after every request. Normally SoapUI will open a http connection and sends the web service requests one by one. However due to few unknown problems on data size limit I encountered, the test case was failing at some point midway through the test steps. However after turning on the option to close http connection after every request the data setup ran very smoothly. This option can be accessed under File -> Preferences -> Http Settings.

SoapUI - Property Expansion

While profiling one of my web services I was looking for an effective tool for load testing my web service. Initially thought of JMeter, but due to my happy experience with SoapUI in the past I decided to use SoapUI.

My request xml had multiple fields however couple of fields I had to send the same data. This wouldn't have been a problem during unit testing, but while load testing this service the value of these fields should be unique per test request. So my problem was say If I am sending 50 requests then in each request the value of these fields though common should be unique. Example request would be as below...

<soapenv:Envelope xmlns:soapenv = "http://schemas.xmlsoap.org/soap/envelope/" xmlns:web = "http://www.xyz.com/webservices/">
<soapenv:Header/>
<soapenv:Body>
   <web:MigrateOrdersRequest>
       <web:Header>...</web:Header>
       <web:Credentials>...<web:Credentials>
       <web:Order>
           ...
           <web:OrderNumber>PO10001</web:OrderNumber>
           ...
           <web:OrderLines>
               <web:OrderLine>
                   ...
                   <web:SerialNumber>PO10001</web:SerialNumber>
                   ...
               </web:orderLine>
               <web:OrderLine>...</web:OrderLine>
           </web:OrderLines>
       </web:Order>
   <web:MigrateOrdersRequest>
</soapenv:Body>
</soapenv:Envelope>

Here I have to pass both PO number and Serial Number unique and same. I initially looked for groovy script, but didn't work out for me. The other feature which really impressed me was Property Expansion which really did it. The property expansion will evaluate the property for you in the soap request may be it is from implicit objects, properties or any random math expression.

The solution initially looked tough was very very simple in the end. I just created a new TestCase in my soapUI testSuite with just one test step which was the soap request. And I created a load test step in the same test case (visibly load test will have only one test step). I used simple strategy with limit as 5 seconds and Test Delay of 5000 milliseconds and giving as many threads as required to mimic the scenario where 'n' (say 50) test requests hit the server. With other strategies where Test Delay was lesser than limit, the counter was not actually unique as the same thread after Test Delay was creating a new request. The modified request is as below.

<soapenv:Envelope xmlns:soapenv = "http://schemas.xmlsoap.org/soap/envelope/" xmlns:web = "http://www.xyz.com/webservices/">
<soapenv:Header/>
<soapenv:Body>
   <web:MigrateOrdersRequest>
       <web:Header>...</web:Header>
       <web:Credentials>...<web:Credentials>
       <web:Order>
           ...
           <web:OrderNumber>PO1000${=context.getProperty("ThreadIndex")}${=context.getProperty("RunCount")}</web:OrderNumber>
           ...
           <web:OrderLines>
               <web:OrderLine>
                   ...
                   <web:SerialNumber>PO1000${=context.getProperty("ThreadIndex")}${=context.getProperty("RunCount")}</web:SerialNumber>
                   ...
               </web:orderLine>
               <web:OrderLine>...</web:OrderLine>
           </web:OrderLines>
       </web:Order>
   <web:MigrateOrdersRequest>
</soapenv:Body>
</soapenv:Envelope>

Here after every load test the PO1000 needs to be modified to give next set of unique numbers. The catch here is that ${=<expression>}was actually evaluating to a value by soapUI at runtime. This expression can be any expression with the objects available in the context like properties, implicits etc.

Database design - basics

Working