Hogwarts is a magical place born from J. K. Rowling’s fantastic imagination. It is also a model at the Warner Bros studio. But did you know, claiming having magical powers could send individuals to prison until the start of the millennium, in the UK.
The ability of producing magic is often confused with supernatural powers. The latter is related to any powers that is outside nature. It has been suggested to possibly heal people, but mostly it remains as a dark art and lack of understanding.
Transform data into knowledge and knowledge into wisdom may sounds like a bit transforming a frog into a charming prince. Frogs are charming and do not needs transforming. Nonetheless, such skills may be perceived by others as witch craft.
The use of mathematical, statistical, and probabilistic knowledge into something useful. Our phone, our web browsers, word processors, and other tools rely more and more on advanced statistical methodologies and machine learning to augment our our intelligence. Using such tools is becoming an expected professional skills in 2024. However, it remains mysterious, use of unnatural powers, and more importantly open doors of misuse that is misunderstood. Large Language models hallucinates and algorithms may not match the data we have at our hands.
To summarise, the world has moved on, and many libraries and other tools have been created to democratise data analytics and data science. Large language models summarise our text, slide design is improved through AI, and image classification is done online. The witchcraft is not understanding what is under the hood but how to use it wisely and carefully.
How can we learn our spell and experience magic?
I am not sure about real magic – I am not a witch. Nonetheless, we can all start to practice data analysis and data science to appreciate better the challenges and what lies beneath the marketing content. Business intelligence tools are considered as insight to support decision making – statistical tests, statistical representation, and predictions. For many the use Quicksigth, Power BI, and Tableau may lead the market, but other open-source tools exists too. All those tools have graphical user interface for the analysis. Learning those tools may be easier. Developing the knowledge and skills to understand what is witchcraft – i.e., unexplainable – or trusthworthy remains important. Therefore, we all need to continue learning to interpret and evaluate the suitability of those findings – and report them to the decision makers.
Jupyter – let’s integrate multiple programming languages
I hope you have not become scared about the terminology programming languages. J.K Rowling appears to have used Latin to come up with her own magic language – the charms and conjuration the language of magic. Even IPhone can become wands. Programming languages are languages to allow us to complete those analysis, and train models (model fitting). They need to be understood and Jupyter notebooks offer ease for all of us to go deeper with our data analysis. Understanding the sequences of clicks and options can be as difficult and challenging to learn – and remember. This first tutorial teaches any witches and wizards to spell their first charms. Beginners will be able to start playing with Jupyter notebooks and start mastering the use of Python.
Is there a platform where we can start our own spellbinding securely?
Data skills can also be pragmatically developed using Kaggle. It is a magical platform where we can learn, develop skills, learn new knowledge from their wonderful datasets, and use pre-trained machine learning models. Our own Hogwarts, where we can all join – without receiving a written invitation.
Kaggle is self-service platforms and free to use. Beginners can learn from the start and build up their skills . No-one never know what they will learn from the large amount of data available. They will save time and effort and try wonderful ideas. You can even participate in competitions. Your progress will reward you.
Kaggle has a lot of data, lots of them. Data science and analysis starts with data. Without data or poor quality data, no one can complete good data analysis or train suitable data models. The data is often kept under strict governance – as it should be. Some data is suitably documented, and some others are not. With that in mind, you can start playing and learning. There is no harm.
There is another side of the coin. The data cannot be fully trusted and the results should be taken with some skepticism. Like Hogwarts, Kaggle is a platform to learn, not to publish or rely fully for any professional outcome. We can only assume that competitions with well-curated data have a verified source.
Anyone can upload some data, but it should break any data protection, copyright, or commercially-sensitive data. It would be ill-advise to upload data from gathered from any professional activities – without any authorisation. It should not be considered.
Witchcrafting rating
The author relies on this platform for her teaching, demonstrating techniques, and developing further her skills. We never stop learning. Her findings may not be significant or suitable for any serious consideration. Nevertheless, some intriguing knowledge have been discovered. It would be reasonable to remain skeptical and find a rating to indicate the level of witchcraft in this work.
A high level of witchcraft is the lack of explainability and validation – mathematically or domain knowledge. A lower one is the reverse; we can explain it. However, it will never be set to 0. There will always be doubt.
Witchcraft rating – 4 leaves
We applied some techniques with various datasets. We start with a high rating of witchcraft. The predictions of whether an athelte can be predicted to win an Olympic medal. The notebook provides all details of the analysis. However, it is yet explained how and why it can predicts 83% of gold medalists, 79% of silver and bronze medialist accurately. The author surmises that the high level of athletes participating without any medals is misleading.
Notebook: https://www.kaggle.com/code/patriciaryserwelch/predicting-medals
Witchcraft rating : 3 leaves
The notebook below explores the use of various time series predictive models with a large data sets from the land registry from the United Kingdom. While the source has yet to be fully verified, the data appears to be complete. However, the data is complex, prices varies a lot. The techniques appears not to be appropriate. The use of index to reduce the complexity appears to be unsuitable. Prices of semi-detached houses in Cambridge, United Kingdom, are likely to be a lot more expensive than predicted. Also, it is unlikely to be linear.
Notebook: https://www.kaggle.com/code/patriciaryserwelch/forecasting-three-uk-cities
Witchcrafting rating : 2 leaves
People lives in cities, villages and country side. However, it would be interesting to explore whether an area of a geographical area is likely to impact the estimated population. The datasets is understood to use census data and other official resources from the United Kingdom. It has been discoverd five clusters may be present. Nonetheless, the clusterisation algorithms fails to fit large areas with low population. The notebook has been commented positively by other. Therefore it deserves a low witchcrafting rating.
Notebook: https://www.kaggle.com/code/patriciaryserwelch/uk-population-and-areas
Witchcrafting rating : 1 leaf
The same analysis focused on the population density. It was assessed both analysis complemented each other. It was interesting to find out the area and population density may not have a strong relationship, but an area and estimated population may have a much stronger one. A clusterisation between the density of population per square km and an area appears to a lot more accurate. Therefore, we could interpret that the area may impact on the concentration of individuals. Other factors needs considering.
Notebook: https://www.kaggle.com/code/patriciaryserwelch/uk-population-and-areas
Are you ready to learn spell binding?
You will see the notebooks uses again simple code. Example code can be found and used again. Learning about the techniques can be found from many books and websites. Data can be transformed into knowledge and knowledge into wisdom. Learning the basic behind those wonderful AI tools, then it becomes much easier to value with scepticisms their outcomes. Usefulness with caution.
Nobody knows what we can learn and discover, when we start our journey into data and in Kaggle.
Interesting Computing 