A practical IdP evaluation framework built from real enterprise requirements. Covers protocol depth, migration, multi-tenancy, AI-readiness, and the criteria most checklists miss.
Most identity provider comparison articles are written by identity providers. Shocking, right? They list features their product has, skip the ones it doesn't, and call it an "objective guide."
This isn't that.
We've reviewed dozens of real enterprise evaluation requests — the actual spreadsheets and RFP documents that procurement teams send to vendors. The patterns are clear: engineering teams consistently underweight the criteria that matter most and overweight the ones that matter least.
The result? Teams pick an IdP based on a demo, discover the migration story is a nightmare six months in, and start evaluating again.
Here's the evaluation framework we wish someone had given us before we started. It's built for engineering teams at B2B SaaS companies — the ones building products, not the ones buying workforce SSO for their employees.
If you're skimming, here's the short version:
The rest of this guide walks through each evaluation dimension in detail, with specific questions to ask and red flags to watch for.
This is for you if:
This is NOT for you if:
Every IdP on the market will tell you they support OAuth2 and OIDC. That's table stakes. The real questions are about depth.
Must-have:
Increasingly critical:
Here's a question most teams don't think to ask: Can you use this IdP as an OIDC Provider — not just an OIDC consumer?
Why it matters: As your SaaS grows, partners and customers may want to use your identity system to log into their own tools. You need to issue tokens, manage consent, and handle third-party app registrations. If your IdP only lets you consume external identity providers but can't act as one, you'll hit a wall when you need to federate outward.
Ask:
The token is the contract between your IdP and your services. If you can't customize it, every downstream service needs to make additional API calls to figure out what a user is allowed to do.
Ask:
A token that carries { "org_id": "org_123", "role": "admin", "auth_level": 2 } means your API middleware can make authorization decisions in one line. A token that only carries { "sub": "user_456" } means every service needs to call back to the IdP or a database to figure out the rest. At scale, that difference is the difference between 2ms and 200ms per request.
Every IdP supports email/password and social login. Congratulations, you've narrowed the field to... all of them.
The differentiation is in the details most demo scripts don't cover.
This is where evaluations get separated from demos. Session and token management is boring until it breaks — and when it breaks, your entire user base gets logged out simultaneously.
Not glamorous. Absolutely critical.
app.yourproduct.com and your API on api.yourproduct.com, can sessions span subdomains? How?Your users want to stay logged in for weeks, not minutes. But a 180-day persistent session has very different security implications than a 30-minute session.
Ask:
Role-Based Access Control is the baseline. If the IdP doesn't support RBAC, it's not worth evaluating. But for B2B SaaS, RBAC alone isn't enough.
Your users belong to organizations. Their permissions within each organization are different from their platform-level permissions.
A user might be an admin in Org A and a viewer in Org B. The same user, two different role contexts. If the IdP can't model this natively, you'll build a parallel permissions system in your application — and now you have two sources of truth.
Questions:
For financial applications, healthcare, or any product where certain operations carry higher risk: not all authenticated sessions are equal.
Viewing a dashboard? Session cookie is fine. Initiating a wire transfer? You need a fresh MFA verification, even if the user is already logged in.
This is step-up authentication, and it requires the concept of an authentication level (auth_level) as a first-class citizen in the token system.
Ask:
auth_level claim that your backend can check?auth_level have its own expiration, independent of the session?If your IdP doesn't support this natively, you'll end up building it yourself — which is exactly the kind of identity logic you're buying an IdP to avoid.
The ideal: your API middleware reads the token, sees the user's org, role, and auth level, and makes the authorization decision without hitting any external service.
The reality with many IdPs: the token tells you who the user is, but you need a separate API call to figure out what they're allowed to do.
That separate call adds latency, creates a dependency, and introduces a failure mode. At 1,000 requests per second, you don't want your authorization check making network hops.
Here's a statistic nobody likes to talk about: most IdP evaluations end not because the new IdP isn't good enough, but because the team can't figure out how to migrate their existing users.
If you have 100K+ users, migration isn't a "nice to have" — it's the entire project.
1. Bulk import with existing password hashes
Your users have passwords hashed with bcrypt, argon2, or whatever your current system uses. Can the IdP import those hashes directly and verify passwords against them?
If yes: users log in with their existing password, and nothing changes from their perspective. Best case scenario.
If no: every user gets a "reset your password" email. You will lose 30-50% of your user base in the migration. This is not hypothetical — we've seen it happen.
2. Progressive (lazy) migration
Instead of migrating all users at once, you migrate them one by one as they log in. The first login hits your old system, verifies the password, and creates the user in the new IdP. Every subsequent login goes directly to the new IdP.
This is the safest approach for large user bases, but it requires the IdP to support:
3. Dual-write (running systems in parallel)
During transition, both the old and new identity systems are active. Writes go to both, reads gradually shift to the new system. This provides a rollback path but adds operational complexity.
If the vendor can't answer these confidently, they haven't done this before. Move on.
B2B SaaS means multi-tenancy. Your customers are organizations with multiple users, roles, and access policies. The IdP needs to understand this natively.
Some IdPs don't have a native organization model. They suggest using custom user metadata (user.app_metadata.org_id = "123") as a workaround.
This falls apart fast:
If the vendor says "you can model organizations using our metadata fields," that's the identity equivalent of storing relational data in a JSON column. It works until it doesn't.
Twelve months ago, "AI agent authentication" wasn't on any evaluation checklist. Today, if you're building AI features into your product — copilots, autonomous agents, AI-driven workflows — your IdP needs to handle a new type of identity: the agent.
Traditional auth has two actors: the user and the application. OAuth2 was designed around this.
AI agents introduce a third: a non-human entity that acts on behalf of a user, with constrained permissions, and needs its own audit trail.
Token Exchange (RFC 8693): The agent presents its own credential plus the user's authorization, and receives a scoped token. The token carries: who (the user), what (the agent), scope (the permission boundary), and when (the expiration).
Agent as a client type: The agent should be modeled as a proper OAuth2 client with its own client_id, not a hack using API keys or shared user tokens.
Delegated scope management: The user can grant specific permissions to the agent — read but not write, access to certain resources but not others.
Audit distinction: Your logs must differentiate between "user did X" and "agent did X on behalf of user." If you can't distinguish these, you'll fail your next SOC2 audit when the auditor asks "who made this change?"
MCP is becoming the standard protocol for AI agents to interact with tools and services. If your IdP supports OAuth-based authentication for MCP servers, agents can authenticate properly through the protocol layer rather than through API keys or shared secrets.
If the vendor hasn't thought about this, they're building for 2020. You're planning for 2026.
Features sell. Operations decide whether you renew.
After evaluating across all dimensions, you need a way to compare. Here's a priority framework:
| Criterion | Why it's P1 |
|---|---|
| Password hash import or progressive migration | Can't use it if you can't migrate |
| Authorization Code + PKCE support | Security baseline |
| Native organization model | B2B SaaS requirement |
| SOC2 Type II or clear path to it | Enterprise customers will ask |
| 99.9%+ uptime SLA | Auth down = product down |
| Criterion | Why it's P2 |
|---|---|
| Custom JWT claims | Avoids per-request permission lookups |
| Per-org authentication policies | Enterprise customer onboarding |
| Organization-scoped roles and tokens | Multi-tenant authorization |
| Refresh token rotation and revocation | Security best practice |
| Hosted UI + custom UI option | Flexibility for different use cases |
| Criterion | Why it's P3 |
|---|---|
| Token Exchange (RFC 8693) | AI agent authentication |
| OIDC Provider capability | Partner federation |
| Step-up authentication / auth_level | Financial or high-risk operations |
| SCIM provisioning | Enterprise customer directory sync |
| Passkey / WebAuthn support | Passwordless direction |
| Criterion | Why it's P4 |
|---|---|
| Built-in analytics dashboard | Can build from audit logs |
| White-labeled email templates | Convenience feature |
| Visual flow builder | Convenience feature |
| Pre-built social connectors (beyond top 5) | Long tail providers |
How to use this: Start with P1. If a vendor fails any P1 criterion, stop evaluating them. Then score P2 and P3 categories. The vendor with the best combined P2+P3 score is your answer.
We've seen teams make the same mistakes repeatedly. Here's how to avoid them:
A feature comparison table tells you what exists. It doesn't tell you how it's built. An IdP might "support" organizations by storing org IDs in user metadata. That checks the box on a spreadsheet but creates real problems in production.
Fix: For every feature, ask "how is this implemented?" — not just "do you have this?"
Teams pick the "best" IdP, start implementation, and then realize they can't migrate their users without a password reset campaign. Now they're either stuck with a bad migration experience or starting the evaluation over.
Fix: Make migration capability your first filter, not your last.
Every vendor's demo is polished. It shows the happy path with a clean database and zero edge cases. Your production environment has users with merged accounts, weird unicode in profile fields, and sessions that shouldn't exist but do.
Fix: Ask for a proof-of-concept with your actual data. Import 1,000 real users and run your actual authentication flows.
If only the platform team evaluates, they'll pick what's technically cleanest. If only product evaluates, they'll pick what's easiest to integrate. If only security evaluates, they'll pick what has the most compliance checkboxes.
Fix: Evaluation team should include platform engineering, product, and security. Each owns different P1/P2 criteria.
Vendor lock-in is real. Proprietary SDKs, custom APIs, non-standard token formats — they all make migration harder later.
Fix: Prefer IdPs that use standard protocols (OIDC, OAuth2, SCIM). Your future self will thank you.
For a thorough evaluation including proof-of-concept testing, expect 4-8 weeks. Rushing it leads to the mistakes we outlined above — particularly the migration oversight. Budget 2 weeks for requirements gathering, 2-3 weeks for vendor evaluation and PoC, and 1-2 weeks for stakeholder alignment.
It depends on your stage. If you have fewer than 10,000 users and no enterprise customers, a lightweight auth library might be fine. But once you need SSO, multi-tenancy, MFA, and compliance documentation, the maintenance cost of homegrown auth exceeds the cost of a dedicated IdP. We've seen engineering teams spend 2-3 full-time engineers maintaining custom auth systems — that's $300-500K/year in opportunity cost.
Customer Identity and Access Management (CIAM) is what your product's end users interact with — sign-up, login, profile management. Workforce IAM is what your employees use to access internal tools (Okta for your company's Slack, Google Workspace, etc.). They're different buying decisions with different evaluation criteria. This guide is about CIAM.
Open-source IdPs offer transparency (you can audit the code), portability (self-host if needed), and community contributions. Proprietary IdPs may offer more polished UIs and managed services. The key question isn't "open vs. closed" — it's "can I leave if I need to?" Open-source solutions tend to make leaving easier because the data model and APIs are public.
If you're already building AI features that access user data on behalf of users (copilots, automated workflows, AI assistants), move it to P1 now. If AI features are on your 6-12 month roadmap, keep it at P3 but weight it heavily. If AI isn't on your radar, it can stay P4 — but revisit in 6 months.
Most IdPs price by Monthly Active Users (MAU). But "MAU" definitions vary — some count any login, others count unique users, some count M2M tokens separately. Get the vendor to quote your specific scenario: X users, Y organizations, Z M2M connections, with your expected authentication volume. Compare total cost, not unit price.
Choosing an identity provider is an infrastructure decision, not a feature decision. You're committing to a system that will handle every user's first interaction with your product, every permission check in your API, and every audit log entry your compliance team reviews.
The evaluation framework above covers what actually matters — not the marketing bullet points. Use it to filter candidates quickly (P1 criteria first), evaluate deeply (P2 and P3 criteria with proof-of-concept testing), and make a decision that holds up for years, not months.
The teams that get this right share one thing in common: they treat identity as infrastructure, not as a feature to ship and forget.