Wednesday, November 6, 2013
Evaluating a Clinical Analytics and Data Warehousing Solution
The healthcare analytics market is a crowded and chaotic one, with many vendors lining up to claim they can help providers use their data to improve care and lower costs. Making a wrong decision at this time in the market will paint you into a decision corner that could last 4-5 years, at least, while the market rapidly moves past your organization towards accountable care and quality-based payment models. You cannot afford the time to recover from a bad decision. So how do you cut through the noise to find a solution that makes the most sense for your organization? This blog is designed to help you with that decision.
General Criteria for Assessing a Healthcare Analytics Vendor
Embarking on an assessment with the knowledge of key, general criteria can help you determine whether a vendor has the philosophy, experience and viability that can lead to a successful outcome for your organization. By evaluating vendors according to the following, you can narrow down the list considerably.
Completeness of Vision
What lessons does the vendor bring from the past healthcare analytics market and how have they adjusted their current strategy and products accordingly? Can they bring lessons from other industries that are more advanced in their adoption of analytics? What is the vendor’s understanding of the present market and industry requirements? What is their vision of the future for healthcare analytics? Look for vendors who can clearly outline how they have evolved to meet—and anticipate—industry needs.
Culture and Values of Senior Leadership
It’s no cliché, but rather the precise truth: the overall culture of a company starts at the top. Get to know the senior leadership of the vendors you are evaluating. Insist on meeting several members of their executive team. Simply put, do the culture and values of a vendor’s senior leadership align with yours? When interacting with individual members of the vendor’s team, ask yourself, “Would I be excited to hire this person into our company?” If the answer is consistently no, find another vendor. More than technology is required to leverage analytics to drive real, sustainable change. Cultural transformation will be required throughout your organization. If your culture and values don’t mesh with that of your vendor, you will encounter significant roadblocks to success.
Specific Experience & Ability to Execute
Nothing is more important than the vendor's specific, related experience to the problem you are trying to solve, and their track record. The HealthCare.gov web site fiasco is a classic example of a vendor who had general experience building web sites, but not specific experience of the type required by the project. Make sure the vendor has a track record for delivering value and satifaction to their clients, on projects that look and feel precisely like yours. What do KLAS, Gartner, Chilmark, and the Advisory Board have to say about this vendor? Find at least three, preferably five, referenced accounts. Do not accept referenced accounts that are pre-screened and selected by the vendor—every client of the vendor should be open to serve as a reference. Do the vendors you are considering have solid referenced accounts that are similar in size and demographics to your company? Ask these references very simply: How satisfied are you with the vendor’s products, services, and overall value? Would you hire them again?
Technology Adaptability and Supportability
The reality is, in today’s connected world, all businesses, including healthcare, move at the speed that their sofotware can adapt-- either fast or slow-- to new processes and business models. Therefore, the underlying engineering and architecture of that software is critically imporant. You must peel back the covers of the vendor’s products and evaluate their software engineering for modern design patterns like object oriented programming, service oriented architectures, loose coupling, late binding, and balanced granularity of software services. Glossing over this assessment is akin to buying a multimillion dollar office building without assessing the modularity of the walls and soundness of the foundation, plumbing, and electrical systems. How fast can the system adapt to the market and your unique needs for differentiation? Data standards, vocabularies and analytics use cases are changing rapidly in the healthcare industry, literally everday, with no signs of slowing down. Find a vendor whose software engineering can keep up. Analytic agility is critical. Executives in your organization can’t wait weeks and months anymore for a new report to inform a critical decision. The industry is changing too fast.
Total Cost of Ownership
The best solution in the world is of no value if it’s not affordable. To assess affordability, you must understand the total cost of the vendor’s solution. Measuring Total Cost of Ownership (TCO) is easy—add up the three-year labor costs, licensing fees (including third party), support fees, and hardware costs associated with a vendor’s solution. Many old-school analytics vendors require a significant upfront investment with no guarantee of value for two years or more. Your TCO over three years should be evenly distributed, not front-end loaded, and your contract should be structured with escape clauses if the vendor’s solution cannot prove value in the first year. In today’s market, clients should expect initial value from analytics vendors in less than six months, preferably three. If a vendor cannot or will not commit in their contract to this timeframe for delivering value, look for another vendor.
The key question here is: Will the vendor be around in nine years (the average life span of a significant IT investment)? If not, can you live without them? Take advantage of evaluations by neutral third-party analysts like Gartner, Chilmark, KLAS and The Advisory Board. What are these analysts saying about the vendor’s prospects in the market? Is the vendor in solid financial shape? What’s their monthly burn rate vs. income? How many days cash-on-hand do they maintain? What’s their sales pipeline look like? Does the vendor’s executive leadership team have a track record for jumping from one company to another or do they have a track record of longevity and success? How much is the vendor spending on sales staff in comparison to engineering and product development staff? The best products are supported by a very lean sales staff—great products sell themselves.
Technology and Cultural Change
Technology is vital to the success of an analytics initiative, but it is only one part of the solution. The meaningful use of analytics is one of the most difficult things for organizations to achieve, culturally. A successful analtyics implementation establishes the technology as well as the sustainable cultural changes required to turn the insights from data into improvements in patient care and reductions in cost.
When evaluating a vendor’s technology, be sure to look at the following:
Data Modeling and Analytic Logic
Different vendors’ analytics solutions feature different data models. Which data model they use can have a significant effect on the cost, scalability and—especially-- the adaptability of your analytics solution to support new use cases. Rapidly adaptable and very flexible, a bus architecture is the best data-modeling option for healthcare. Most vendors utilize a healthcare-specific enterprise data model at the heart of their solution, but these enteprise data models are difficult to load and map initially, and slow to evolve subsequently, when faced with new use cases and source system data content. Over-modeling data is the single most significant contributor to data warehouse and analytics failure in healthcare. My advice is simple: Stop modeling your data and start relating it. Relating data is what analytics is all about. These enterprise data models come in three basic flavors, so be aware of them. They are: (1) Dimensional star schema; (2) Comprehensive, enterprise data model; and (3) I2B2.
In addition to the issue of data modeling, the analytic logic associated with the content of data marts and reporting is critically important. To learn more about the role of data modeling and “binding” data to business and clinical logic in healthcare analytics, read this white paper: The Late Binding Data Warehouse.
Master Reference/Master Data Management
The ability to incorporate data from new and disparate sources into your analytics solution requires significant expertise in master data management. What is the vendor’s strategy for managing unique patient and provider identifiers? How does the vendor accommodate international, national, regional and local master data types and naming conventions? Do they support mappings to RxNorm, LOINC, SNOMED, ICD, CPT and HCPCs? How tightly does the vendor bind your data to the vocabularies that change regularly? The tighter the binding, the less flexible the analtyic design will be to accomodating changes in the vocabulary and analytic use cases based on those vocabularies.
An effective metadata repository is the single most important tool for the widespread utilization and democratization of data in an organization. Look for a vendor that provides a tightly integrated, affordable, simple repository with their overall analytics solution. The most valuable content in a metadata repository is not computable—the most valuable content is subjective data that comes from the data stewards and analysts who have interacted most with the data. Look for vendors that have the ability to maintain this subjective data through a wiki-style, wisdom of crowds contribution model. A web-searchable metadata repository should provide information such as the source of the data, how often it is updated, examples of the data, natural language descriptions of the physical data tables and columns, any known data quality issues, and the contact information for the associated data steward. The ability to quickly establish the origins and lineage of data in a data warehouse is also a critical component to an effective repository. Analytic vendors tend to operate in one of two extremes: (1) They either oversell very complicated and expensive metadata repositories that require an overwhelming level of support and maintenance in return for a declining return on investment; or (2) They offer no solution for metadata management, which is disasterous to a long term analytic strategy. Find a vendor that offers a simple, low cost, pragamatic solution between these extremes.
Managing “White Space” Data
Does your analytics solution offer a data collection alternative to the proliferation of desktop spreadsheets and databases that contain analytically important data?
White space data is the data that is collected and stored in desktop spreadsheets and databases that it is not being collected and managed in primary source systems, especially EMRs, or it is being collected in clinical notes and must be manually abstracted for reporting and analysis. This desktop data fills in the missing “white space” of analytic information that is important to the organization. For example, these desktop data sets are commonly found in support of Joint Commission reporting, internal KPIs, finance analytics, and clinical researchers. It is not unusual for healthcare organizations to have hundreds of these desktop data sources that are critically important to the analytic success of the organization. However, because the data resides on desktop computers and shared drives, it cannot be integrated with other mission critical analytic data that is being stored in the Enterprise Data Warehouse from the primary source systems. Data synergy suffers as a result. White space data also poses information security risks. Analytics vendors must provide a tool for attracting the management of white space data into the content of the EDW. Look for a white space data management tool that is web based, as easy to use as a spreadsheet or desktop database for the collection of data, and makes is easy for end users to convert and upload their existing desktop data sets. Also, look for a security model in the EDW that allows for the isolation and stewardship of these white space data sets.
The best analytics solutions include a bundled visualization tool, and this tool should be affordable and extensible if licensed for the entire organization. However, the analytics visualization layer is very volatile. The leading visualization solution today will not be the leader tomorrow. Therefore, look for an analytics vendor that can quickly and easily decouple the underlying data model and data content in the data warehouse from the visualiztion layer, and swap the visualization tool with a better alternative when necessary. Also realize that a single visualization tool will not solve all of your organization’s needs. Data analysts will want to use a variety of tools to access and manipulate data in the enterprise data warehouse. The underlying data models in the data warehouse must be capable of supporting multiple visualization tools at the same time. Ask vendors if their data model is decoupled from the visualization tool. Does the data model support multiple visualization tools and delivery of data content?
As always in healthcare IT, the privacy and security of patient data are paramount. Here are some important questions to ask a potential analytics vendor about security:
· Are there fewer than 20 roles in the initial deployment? Contrary to popular belief, more roles can actually lead to lower security and will definitely lead to higher overhead adminisrative support costs.
· Does the solution employ database-level security, visualization-layer security or some combination of both? The vendor’s solution should support both.
· What is the vendor’s model for protecting patient identifiable (protected health information (PHI)) data and the more sensitive subsets of PHI that are typically defined at the local State-level, such as mental health data, HIV data, and genomic/familial data?
· What type of tools and reports are available for managing security and auditing access to patient identifiable data?
A robust ETL process—how analytics technology extracts data from source systems, applies the required transformations and writes data into the target database—is fundamental to the success of your chosen solution. Ask vendors to demonstrate how their ETL measures up in terms of reliability, supportability and reuse. At present, Microsoft’s ETL tool—SQL Server Integration Services (SSIS)—is by far the most cost effective ETL tool in the market, offering the highest value per dollar.
Performance and Utilization Metrics
As you implement and continue to use an analytics solution, you will need to generate metrics about who is using the system, how are they using it, and how well the system operates. Can the vendors’ solution track basic data about the environment, such as user access patterns, query response times, data access patterns, volumes of data and data objects? This kind of information will be essential to you as you refine and organize the data content and analytics services you provide from the data warehouse.
Hardware and Software Infrastructure
Does the vendor use Oracle, Microsoft or IBM for its hardware and software infrastructure? These three are the only viable options in today’s healthcare market and data ecosystem. Hadoop and its associated open source tools is not an appropriate analytic and data warehousing infrastructure option at this time in healthcare (accept for gene sequencing). Microsoft is the most integrated, affordable and easiest to manage of these technology platforms and now makes up 70% of all new sales in the analytics and data warehousing market, across all industries, far outpacing Oracle in new sales, it’s closest competitor. Rumors about Microsoft’s ability to scale up to large, multi-terabyte data warehouses are totally unfounded. Microsoft’s parallel data warehouse platform can scale to the petabyte level, far beyond the largest data warehouse needs in the healthcare provider space. Ask any data engineer that has worked extensively on either Microsoft or Oracle, “Which platform is the easiest to use and most efficient for delivering quick, adaptable analytics solutions?” Their answer will almost certainly be Microsoft.
Cultural Change Management
As mentioned previously, technology is important—but it is only a part of the equation in creating a successful analytics program. A vendor’s solution must also include processes and real-world experience for helping your manage sustainable change in your organization, driven by analytics. Nothing is more politically or culturally disruptive than the spotlight of analytics, not even the deployment of an EMR. You want a vendor that has been there, in the trenches of cultural transformation driven by the enlightment of data.
The Healthcare Analytics Adoption Model
The model outlines eight levels of analytics adoption an organization passes through as it gains sophistication in using its data to drive improvement. Like a course curriculum for college studies, following this model with discipline will lead to the successful adoption of analytics in your organization, culturally and technically. Use this model for evaluating vendors’ capabilities in each level—have the vendor demonstrate their products and services for each level. The model also provides a roadmap for your organization to measure your progress of analytics adoption. Ask yourself, “How fast do we want to achieve the highest levels of adoption in this model?” With the right analytics vendor as a partner, organizations can achieve Level 5 within 18 months of implementing and following the model, and some organizations can make it in 12 months. Level 7 is easily achievable within 24-30 months.
This model provides a standard to help you move beyond a patchwork of point solutions to a robust, data-driven health delivery system capable of tailoring care while optimizing efficiency and cost. Does a vendor’s solution support each level in the model? Ask them to prove it.
The details behind this model, including a self-inspection guide, can be found at: Healthcare Analytics Adoption Model
Healthcare is at the threshold of the next revolution in data management—being able to analyze and make informed decisions based on the data that organizations have been collecting and sharing. This is a critical time to set your organization on a pathway for data-driven improvement. The criteria outlined in this paper will help you choose an analytics partner with the expertise, processes and technology to help you achieve this objective.
Nuclear warfare operations was my data-driven decision making environment before the healthcare phase of my career. It was all about recogni...
Amazon.com announced today that it will develop a new Electronic Health Record based upon the same user interface experience and underlying ...
For the last couple of years, I've been using this metaphor to highlight the ridiculous nature of healthcare and particularly...
These are the five books that our 4-year old daughter, Anna regularly pulls from our bookshelves to "read." Why she chooses them, ...