I'll refer to "I" and "J" here for a generic vowel and corresponding semivowel. Here are some tips to cut down the list of phonemes when they aren't all really needed. Note that /A/ > [B] means "phoneme (class of sounds) A is realized as phone (specific sound) B".

• If you don't contrast [I] and [J], just use one phoneme. This applies in general.
• If you have something like /JI/, it might not be [JI] but instead [JƏI], where Ə is some more central vowel. Whether you want to adapt the phonology to represent this or not is up to you.
  • Example: "woo" in my dialect is [wʊu].
  • Example: "yee" in my dialect is [jɪi].
• If [IV] never happens but [JV] does, let /IV/ > [JV]. Similarly for [VI] and [VJ].
  • Example: /uæu/ > [wæw]
• If just one of those cases does happen, try a pair of rules like /IV/ > [JV] and /IIV/ > [JV]. This has the effect of treating [I] as a geminated /J/.
  • Example: /tia/ > [tja], /tiia/ > [tia]
• If both, try /IVV/ > [JV] and /IIV/ > [IV].
  • Example: /tiuu/ > [tju], /tiiu/ > [tiw], /tiu/ > [ti.u]

Sorry if this is a little confusing. If you have questions about a specific phonology, maybe I can make this a little clearer. A lot of these problems come down to your language's phonotactics, since it relies on being able to infer the realization of a phoneme consistently based on its environment, since that's fundamentally what makes a phoneme.

Thanks for reading!