Creditvidya

Read Spark Parquet Output Using Python Job Using Arrow

Jun 10, 2021 · 2 min read · Hadoop BigData Apache Spark Apache Arrow Python Parquet Pandas Dataframe ·

We have a lot of different jobs which run on our Spark Clusters and produce the output in different formats as per the job specifications. Generally the data format that we use are json, parquet and parquet with data partitioning. There is this one case where we wanted the paquet output data generated by one of the …

Encoding vs Encryption vs Hashing vs Obfuscation

Jun 8, 2021 · 4 min read · Encoding Encryption Hashing Obfuscation Security Passwords ·

Introduction Encoding, Encryption, Hashing, Obfuscation all are applied to data for some or another purpose and all of them create significant confusion when to use which one. Lets clear out the confusion so that we can be confident while using these terms in our next tech discussion with our clients/colleagues. …

How The Pandemic Made Infrastructure Leaner

May 31, 2021 · 5 min read · AWS Spot Cost Reduction EBS S3 Cloudformation Cloud ·

At CreditVidya, much like every other tech firm, cost-optimization is the first thing that business leaders rushed to do at the onset of the pandemic-induced economic slowdown. We achieved a cost reduction of 35% by optimizing the infrastructure using Amazon Web Services. As a fast-growing startup working with big …