Computers have powerful built-in components such as CPUs, memory, etc. CPUs can perform millions of instructions per second. How can we use computation and storage capability for our own benefit?
For example, I want to process payroll data for thousands of employees to display hundreds of products on an eCommerce website. Programming language helps us give instructions to computers to perform some activities. You might have heard about Java, Python, C, C++, PHP, COBOL, .NET, BASIC as they are examples of high-level programming languages.
We all know that computers can understand only electronic signals. There are only two possible things computers know: The presence of a call (one) and the absence of a signal(zero). Computers use machine code (Contains zeros and ones) for processing purposes. Now how do we give instructions to computers when they don’t understand English, Spanish, or French.
Software professionals came up with the idea of high-level languages using which we can give instructions to CPUs to perform some activity. High-level languages are built with simple keywords from plain English and convey special instructions to CPUs for each keyword. Therefore, we write a set of statements in any high-level language to perform some computational activity.
This set of statements(code) is called a program. In the software field, this activity is known as coding or programming. With that background information, we will see one more fundamental concept.
What is Python?
Python is a general-purpose programming language. If you see any survey on popular programming languages over the last few years, Python always comes on top of the demand chart each year.
People who hate programming are tempted to change their minds by the simplicity of Python. Most of the job listings have Python somewhere as part of the job description. Let us see what makes Python so unique in this decade and the future. Candidates and companies upskill or reskill in Python irrespective of their qualifications, roles, and experience. Needless to say, Python is straightforward to learn.
It is a high-level language used by millions of developers across the globe. Despite being a highly respected programming language, it is surprisingly readable and can be used by novices and veterans alike.
Python combines various high-tech features such as speed, productivity, community and open source development, extensive support libraries, third-party modules, and more to improve programming. Whether you want to create business applications, games, operating systems, computational and scientific applications, or graphic design and image processing applications, python has got you covered. Learning it increases your chances of landing lucrative gigs and joining the bandwagon of celebrated cloud computing experts.
Python is globally well supported and adopted by the tech community
Some languages are designed to solve specific sets of problems. For example, Structured Query language (SQL) is meant for working with databases. LISP favoured Artificial Intelligence related research, and FORTRAN were developed for scientific and engineering applications.
Python is a general-purpose language. You could use Python in different domains such as the data science field, web development, application development, game development, information security field, system administration, image processing, multimedia, IoT, and machine learning.
Suppose you want to know whether Python has advanced features of a standard high-level language. Yes, it does. Python supports important language features: dynamic typing, late binding, garbage collection, exception handling, and more than 200,000 packages with a wide range of functionality. Python is very stable and has been here for three decades.
Python also supports different programming approaches. For example, Python supports structured programming, object-oriented programming, functional programming, and aspect-oriented programming.
Python is free and has strong support from the global Python community. A non-profit organisation, the Python Software Foundation manages and directs resources for Python. Stackoverflow developer survey results say that Python is the most popular and wanted general-purpose language.
The trump card of Python is that it is one of the easiest languages to learn and read. Even though Python has been around for nearly 30 years, it has evolved considerably, which is the main reason why it has gained popularity with software engineers.
However, the main reason why Python has held its ground is that, like Java, it is a universal programming language that can be used in many areas, including cloud computing. The following are some of the main features that make it highly compatible with cloud computing.
- Can be used in computational and science-based applications
- It can be used in image processing and design applications
- Used in GUI-based desktop applications
- Can create operating systems
- Used widely in computer games
- Used in web frame applications
- Applied in business applications
Cloud service providers make use of specialised packages and tools for coding in different languages. Python can be used on these cloud computing infrastructures. On IBM Bluemix, the cloud services can be accessed using a console and can be used once the credentials are verified. Cloud infrastructure can be used with the Cloud Foundry CLI (Command Line Interface).
Programming Python for NoSQL databases
NoSQL databases are being used in social media applications and portals that process Big Data in which enormous, heterogeneous and unstructured data formats are handled. NoSQL databases are used for faster access of records from the big data set at the back-end.
The Aadhaar card implementation in India is being done using NoSQL databases as vast amounts of information are involved, including text data, images, thumb impressions, and iris detection. No traditional database system can simultaneously handle data sets of different types (text, video, images, audio, thumb impressions, iris samples, etc.).
Currently, several NoSQL databases are used for different types of portals, and these specialise in handling heterogeneous and unstructured data. In traditional Web-based implementations, RDBMS packages are deployed for database applications including Apache Derby, MySQL, Oracle, IBM DB2, Microsoft SQL Server, IBM Notes, PostgreSQL, SQLite, Sybase, etc.
These are known as traditional SQL databases that comply with the ACID (atomicity, consistency, isolation, and durability) properties. NewSQL is the new-generation database engine that provides the scalability and performance of NoSQL systems for online transaction processing (OLTP) read-write workloads while maintaining the ACID guarantees of a classical database system.
Nowadays, Web applications use unstructured data in heterogeneous formats, including audio, video, text, streaming, signals, images, pixels, and many others. In each file pattern, there are many file formats. For instance, there are some file formats in the video, including MPEG, MP4, AVI, 3GP, WMV, OGG, FLV, and others.
In the same manner, image or graphics file formats include GIF, PNG, JPEG, PCX, BMP, TIFF, and lots of others.
The significant issue here is the compatibility of Web applications with all these file formats in different domains. Here is where the concept of NoSQL databases comes into play since they enable any type of file format to be processed and integrated into Web applications.
A NoSQL (Not Only SQL) database provides the system for storing and retrieving data and is not modelled on the tabular relations methodology used in relational databases. The data structure in NoSQL databases is entirely different from that in traditional RDBMSs. Currently, the former is being rapidly adopted in Big Data and real-time Web applications.
There have been various approaches to classifying NoSQL databases, each with different categories and sub-categories. Because of the variety of techniques and overlaps, it is difficult to get an overview of non-relational databases.
Python is widely used in different application domains
Web and internet development
Python offers many choices for web development:
- Frameworks such as Django and Pyramid.
- Micro-frameworks such as Flask and Bottle.
- Advanced content management systems such as Plone and django CMS.
Python’s standard library supports many internet protocols:
- HTML, XML, and JSON
- Email processing.
- Support for FTP, IMAP, and other internet protocols.
- Easy-to-use socket interface.
Scientific and numeric
Python is widely used in scientific and numeric computing.
Python is a superb language for teaching programming, both at the introductory level and in more advanced courses. Schools and colleges are started to offer Python as a beginner-level course for programming. So there are many openings in teaching as well.
The Tk GUI library is included with most binary distributions of Python.
Software development (DevOps)
Python is often used as a support language for software developers to build control and management, testing, and in many other ways.
Python is also used to build ERP and e-commerce systems. As per TIOBE Index for December 2019, Python is placed in one of the top three languages to build new systems.
Using Google Cloud
Starting out, you’ll be using the online console. It’s the place to create, edit and monitor the resources you’re using for your project, all in a simple visual interface.
Google Cloud organises your account into ‘projects’, which keep your app/scripts/ideas logically separated from each other. Each project can use any number of Cloud resources.
These resources can be things like virtual machines, machine learning APIs, or databases. These are all set up to work out of the box from the console, making it very easy to use with your projects. This is a lot of the appeal with Cloud hosting companies: all the infrastructure and complicated code is done for you and abstracted away, leaving you to focus on the principle that really matters.
Your account can be granted access to projects to varying degrees by creating an IAM account linked to your login and project. Your IAM account can be assigned any number of ‘roles’, which define what you as a user are permitted to do in that project.
For example, you could have the role ‘Editor’, which gives full edit access to a project. You could also give others access to individual resources or services to varying degrees, like editing access to a virtual machine but reading-only access to a database.
The other type of account in Google Cloud is the ‘service account’. You can think of this as an IAM account for your resources themselves, granting them access to other resources. A common use case is letting code on a virtual machine read and write data to a database.
What Actually Runs Your Code?
There are a few ways to run code in Google Cloud. The most simple is the ‘Compute Engine VM Instance’ — essentially a virtual machine. You can customise a VM Instance with options like the size of the processor, amount of RAM, storage size, operating system, and even its geographic location.
Since a VM Instance is just a computer, it can do anything a computer can do: make files, install apps, and run Python. There are other ways of running code (Kubernetes, Cloud Functions, Anthos), but these tend to be for larger applications that require more organisation. And anyway, at the very fundamental level, all of the other methods are just VM Instances with fancy software on top.
Creating Python script to automate
First of all, to automate a script, we must have a script. In this case, we will do something quite familiar: extract data from an API and upload it to a database. It is something very typical that is usually done in ETL processes.
For this, I have created a free database on Heroku. This is a Postgre database and, regarding the automation, the table is already created, we just have to add new data.
In any case, the idea is as follows: we will extract the position (latitude, longitude, etc.) of all the aeroplanes in circulation using the OpenSky API, and we will upload them to our database. This is something that we would have to do, for example, if we want to visualise the evolution of the position of the planes.
Setting up our function in Cloud Functions.
Cloud Functions is a Google Cloud service that allows us to run Python functions in the cloud without setting up a server. Besides, Cloud Function offers two exciting things:
- We can choose on which Google Cloud servers the function will be saved. Being able to select the location of the server will give you a shorter response time.
- The trigger of the function. We have different motivations available, from an HTTP point to a Pub / Sub topic.
In our case, we will choose a Pub / Sub topic as the trigger as it will make automation more accessible and more secure. Let me explain myself: if we decide on HTTP calls, we must choose whether the call will require credentials or not. Ideally, the call should need credentials, otherwise, anyone could invoke it, and we do not want that.
If we set the credentials as required to call that endpoint, we will have to download the credentials of a service account and pass them in the script. Instead, if we subscribe to a Pub / Sub topic, the authentication is done internally, so the credentials are unnecessary. Of course, when automating the function with Cloud Scheduler, we will have to send messages to that topic instead of making POST requests.
Uploading Python script to Cloud Functions
First of all, we must choose the programming language that we are going to use. In our case, Python. After that, main.py will appear. In this file, we must define the function that we want to be executed. In this sense, it is essential to highlight the following:
- Pub / Sub will pass two parameters to the function. Therefore, our function should require two parameters, even if they are not used later.
- For Cloud Functions to work correctly, the entry point and the function name must match.
- Also, to debug and see what has gone wrong, I would recommend doing error and status returns. In my case, I only return an Ok to check that everything is fine.
Finally, we have to define the requirements of our script in the requirements.txt file. Here we must indicate which packages and in which versions our script needs to work. Once we have configured the function and its requirements, we simply have to deploy the process. If the display is correct, we will see a green check with our function. However, even if the green statement appears, this does not mean that your function is working correctly. To check so, you have to go to ‘Testing’ and click on ‘Test the function’.
Automating Python script on Google Cloud is something simple (much more than automating an R script in Google Cloud) but very useful.
In addition, it is not expensive in terms of costs: Pub / Sub offers 10Gb of complimentary messages per month, Cloud Functions 2 million invocations per month, and the first three tasks of Cloud Scheduler are free. In short, for simple processes, indeed, there is no need to pay.
But, despite being free (or very cheap), with what you have learned today, you can take your Python scripts to the next level, whether you need to automate ETL processes or even if you want to put your algorithm into production!