What is Big Data – challenges and solutions

Irish businesses believe that Big Data, analytics and cloud technologies will deliver the most value over the next two years.

EY’s Tech Horizon Report, which explores how technology and transformation can address key global challenges facing senior technology leaders, found that 50% of Irish respondents believe cloud technologies will deliver the most value to their business over the next 24 months.

In addition, 46% of respondents thought data and analytics would add the most significant value.

Real-time access to data has already transformed Irish businesses by empowering them to understand customers better and anticipate their needs.

The Bank of Ireland states that data is the currency of the digital economy and Irish businesses should be quick to capitalise on this to become more competitive and attractive to customers, employees and partners.

The report adds that SMEs may have an advantage in this area because of their size.

But what is Big Data, and why is it important?

big data and cyber security - laptop and networking gear

What is Big Data?

Big Data refers to large, complex data sets.

These data sets are too voluminous for traditional data processing software to manage.

The benefit of Big Data is its ability to address business problems you might not previously have been able to tackle.

The three Vs of Big Data

Industry experts may refer to the three Vs of Big Data and these can be useful in understanding what Big Data is:

  • Volume – Around 2.5 quintillion bytes of information is created each day. It is not uncommon for large companies to have Terabytes (roughly one trillion bytes) and even Petabytes (1,000 Terabytes) of data.
  • Velocity – Refers to the speed data is received and acted upon. Some internet-enabled products operate in real-time or near real-time and may require real-time evaluation and action.
  • Variety – Refers to the many types of data available. Once taking the shape of database files – such as Excel – data now comes in non-traditional forms, such as video, text, pdf, and graphics, and via tech, such as wearable devices. Unstructured and semi-structured, data including text, audio and video, require additional processing.

While Big Data and cloud computing are two distinctly different ideas, they are interwoven and almost inseparable. Cloud computing provides resources and services on demand, with significant Big Data stored in the cloud.

However, there are a number of Big Data issues and challenges that organisations may need to tackle to tackle advantage of the potential of Big Data.

Five challenges of Big Data and their solutions

1. There is so much data to make sense of

One of the three Vs of Big Data presents one of its biggest challenges – volume – and this is a key challenge to address when understanding what Big Data is and how it works.

Data is everywhere. Dell Technologies reported that 70% of Irish Businesses say they are gathering more data than they can use. Over half had increased the volume of data they were collecting, but the majority said they couldn’t analyse all the data they collected.

Businesses typically have data for customers’ interests, such as how many people are visiting their website, subscribing to a newsletter, active on social media channels, and financial information.

It’s a potential data goldmine, so how do you mine it?

Solutions for analysing Big Data

A good data strategy explores data collection, integration and management.

Start by looking at how you’re collecting data. For example, are the systems you use to collect customer information accurate? Such as, are there checks on your website to ensure that customers enter the correct number of mobile phone digits? Or are you checking addresses against a Postal Address Finder?

Secondly, to get the most out of your data, integrate databases. Cloud computing provides a valuable platform for storing Big Data in a single place but it is essential to consider data security.

When analysing Big Data, Artificial Intelligence (AI) is rapidly becoming as intertwined with big data as the cloud.

Many technology providers offer cloud-based AI solutions to help make sense of data.

2. Data protection and data security

Gathering and storing large amounts of data raises the stakes for security breaches, especially if data is less well organised. Organisations may have Big Data security concerns, especially related to Big Data and cyber security.

Learn more about ISO 27017 Cloud Data Protection certification used by organisations that require cloud services and by cloud service providers.

Moving Big Data to the cloud means companies often entrust Big Data or customer data to a third-party provider.

Potential security and data threats may include:

  • Fake data – Collecting data from multiple sources increases the risk of collecting the wrong, incorrect or contradictory information.
  • Unprotected stored data – Storing data without safeguards such as a firewall or encryption can leave it at risk of a data leak.
  • Non-compliance with data privacy laws – The EU’s General Data Protection Regulation (GDPR) is the world’s strictest privacy and security law. GDPR applies if you process the personal data of EU citizens or residents, or offer goods or services to anyone resident in the EU – even if your organisation is located outside the EU. Fines for violating the GDPR are high. Companies that lack a proper strategy for big data have a higher risk of exposure from failing to gather the appropriate consent.

Learn more about how our Cyber Essentials certification can demonstrate your organisation’s commitment to cyber security, and learn more about what Cyber Essentials is and how it can help your business.

Read our guide on how to protect your business from cybercrime.

Solutions for securing Big Data

You can follow a few rules to implement data security, including knowing your data, having tighter controls over your data stores and backup, and safeguarding your network against unauthorised access.

ISO 27001 certification, for example, provides a framework to help organisations of all sizes protect their information systematically and cost-effectively.

big data issues and challenges - technician using networking gear

3. Storing data in silos across an organisation

Data silos are a common problem when dealing with Big Data.

What is a data silo?

A data silo is a store of data that a single department or business unit controls and doesn’t share. Typically, siloed data sits in a standalone system and may be incompatible with other data sets, so it’s hard for others within the organisation to access and use it.

Any organisation that lacks a data management strategy can create data silos.

Negative consequences of data silos include:

  • incomplete data sets
  • inconsistent data
  • duplicate data and processes
  • less collaboration and data sharing
  • data security and compliance issues

Solutions for dealing with Big Data silos

There are various means to break down data silos, including:

  • Consolidate data into a single application.
  • Deploy data warehouses and data lakes to create structured ways of holding large volumes of Big Data.
  • Introduce a comprehensive data management strategy that aligns data management processes with business operations.
  • Review your organisation’s culture and consider a data governance initiative.

Learn more about the security issues in cloud computing with our guide.

4. Your company lacks the skills needed for managing Big Data

Many organisations find recruiting the right people to organise, manage and analyse big data challenging.

Invest in Big Data training for your staff

Organisations implementing big data strategies and solutions should assess their skill requirements regularly to identify potential skill gaps. Filling gaps can be addressed by extra training/cross-training existing resources, hiring new resources, and leveraging consulting firms.

Read our cyber security checklist and essential controls for SMEs.

5. Data access control

In breaking down silos and creating a central store for your Big Data, it’s crucial to have rigorous policies around who can view and edit the data. Cloud-based systems such as Microsoft Teams, for example, are designed to enable collaboration.

How to control access to Big Data

A shift from on-premises to cloud-based solutions means it is easier to control who has access to data by implementing Identity Access Management (IAM).

Technologies include the use of multifactor authentication, including 3D authentication. It requires those accessing Big Data to prove who they are by combining something they know (such as a password) and something they are (such as a fingerprint) with something they have (a linked mobile phone number that receives a One Time Passcode).

Following relevant ISO standards is a good starting place to ensure organisations meet the best IAM practices.

Learn more about how ISO 27001 can help your business manage and secure Big Data.

Get a Quote

what is big data guide - main image
Kim Bradley
Kim Bradley

Keep up to date with our latest news!


Related ISO Certifications

ISO 9001 - Quality Management

ISO 9001

Quality Management System ISO 9001 is the internationally recognised global standard for Quality Management Systems. It confirms an organisation’s commitment

ISO 14001 - Environmental Management System

ISO 14001

Environmental Management System ISO 14001 standard is the global standard for organisations wanting to demonstrate their environmental credentials. It

ISO 45001 - Occupational Health and Safety

ISO 45001

Occupational Health and Safety ISO 45001 is an international standard that specifies requirements for an occupational health and safety

ISO 50001 - Energy Management Systems

ISO 50001

Energy Management Systems ISO 50001 is a global standard for organisations looking to improve their energy management

ISO 27001 - Information Security Management Systems

ISO 27001

Information Security Management Systems ISO 27001 is the international standard for managing risks related to the security

ISO 22301 - Business Continuity Management Systems

ISO 22301

Business Continuity Management Systems ISO 22301 is the business continuity management system (BCMS) standard. It provides a framework that

ISO 20000-1 - IT Service Management Systems

ISO 20000-1

IT Service Management Systems ISO 20000-1 Service Management is the international standard for quality management specifically focused on IT

ISO 13485 - Medical Device

ISO 13485

Medical Devices ISO 13485 is a globally recognised quality standard that identifies the requirements of a quality management system

ISO 27701 - Personal Information Management System

ISO 27701

Privacy Information Management Systems ISO 27701 is the global standard for Privacy Information Management Systems (PIMS), also known as

Personal Information Management System - BS 10012

BS 10012

Personal Information Management System BS 10012 provides a framework for a Personal Information Management System standard, helping you maintain

ISO 27017 - Clour data protection

ISO 27017

Cloud Data Protection ISO 27017 is the global standard used by organisations to strengthen their current cloud data protection

Cyber Essentials - Certification Europe

Cyber Essentials

Cyber Essentials Cyber Essentials is a globally recognised IT security standard developed by the UK’s National Cyber Security Centre, which is

ISO 20121 - Event sustainability management systems

ISO 20121

Event Sustainability Management Systems ISO 20121 is an internationally recognised standard for event sustainability management systems. It provides organisations


Recent Insights