I'm starting to explore EventBridge as our async comms framework and this seems extremely useful!
One thing I've been trying to find out more about is how I could integrate with and leverage the EventBridge Schema Registry? It seems like there's some potential for this library to do so. Another alternative could be to just write a script to generate pydantic code from the published schemas and keep that updated/checked-in. Thoughts?
Sometimes you might want to update secrets from within a Kubernetes Pod. For example, if you have the access token and refresh token for an OAuth application, you might need to update these every time you refresh them. I’m sure there’s a few different ways to go about this, and there’s always the option of just storing these in a DB. In my case, I needed to refresh my tokens for Fitbit and wanted to keep them as secrets in Kubernetes. After reading some stuff online, I found the
kubectl patch command, which seemed to be what I wanted. …
Gitlab has a cool internal culture that makes things public by default. This includes their entire Data Analytics pipeline. I wanted to get familiar with their infrastructure, so I decided to take their work and adapt it for my own purpose.
A while back, I wrote some code to pull data from Fitbit, Trace (snowboard tracking app), and some snowfall data in order to do some data analysis about last year’s ski season. I decided to automate the ingestion of Fitbit and Trace data, and do my transformations using DBT instead of Pandas.
The (simplified) pipeline consists of:
This is the second part of a 2 part post on my Python development. Click here for the 1st part.
The main things I need/want from my IDE are:
I’ve tried using a variety of editors an IDEs, but I always end up finding something I don’t like and come back to Emacs. My main issues tend to be around using different environments (one IDE for Java, one for Python, one for Web development…
Ohh XKCD, always on point :)
The more you do with Python, the more complicated your setup can get. For example, you might want a virtualenv for per project. For some projects, you might want to have a full sand box, and for others you might want to be able to use the system wide python environment. There might be other projects where you might want to test with, or use, multiple Python versions. Some might be Anaconda projects. You might have your own scripts that depend on system-wide packages, but you want those to be accessible from anywhere. You…
I was talking to a friend who is working with CSVs with a lot of columns, and he was asking what the best way to read in only the columns he needs. IMO, the best way would be to store the data in a DB so you can
SELECT only the columns that you need, but unfortunately that’s not an option for him. I also thought about pre-filtering the data using
awk on command line, but for simplicity I think Pandas is the right tool.
However, like everything else in Pandas, there are still multiple ways to accomplish…
At one point, I got carried away making a bunch of user level services for Systemd. It’s a neat way to run scripts during user sessions providing cool functionality like handling dependencies and automatic restarts. One issue I ran into was that I couldn’t create units depending on system level targets. I wanted to have some user level units that depended on the sleep.target, but I was blocked for a while. At some point I was introduced to dbus-monitor, which I immediately realized this could be used to build out monitor services to watch for system level events. I was…
When I first got my headset, I kept having problems where A2DP wouldn’t work right upon connecting to bluetooth. I could get it to work by reconnecting occasionally, but nothing worked consistently. I googled around and found https://askubuntu.com/questions/765233/pulseaudio-fails-to-set-card-profile-to-a2dp-sink-how-can-i-see-the-logs-and, from this thread I found a few different solutions, some which worked. However, I wasn’t happy with manually running some sort of
a2dp-fix script each time I plugged my headset in, so I set out to automate it. The main issue I had was that
bluetoothctl wouldn’t work when running from a udev rule. After looking at some examples and digging into…
I’m Diego Fernández Giraldo, a Freelance Data Science Engineer looking to help your business succeed by ensuring you are leveraging data to its full potential.